Jul 24 00:10:37.051592 ip-10-0-139-9 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.414136    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418119    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418133    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418138    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418141    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418144    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418147    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418150    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:10:37.563883 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418153    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418157    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418160    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418163    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418166    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418169    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418171    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418174    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418177    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418180    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418184    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418187    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418190    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418192    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418195    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418197    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418200    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418202    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418206    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418208    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:10:37.566072 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418211    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418214    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418216    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418219    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418221    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418224    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418227    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418229    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418232    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418234    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418237    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418240    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418243    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418247    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418249    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418253    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418255    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418258    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418261    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:10:37.568493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418264    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418266    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418269    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418271    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418274    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418279    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418282    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418285    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418288    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418291    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418293    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418296    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418300    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418303    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418306    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418308    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418324    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418329    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418332    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:10:37.570275 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418335    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418338    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418341    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418344    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418348    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418351    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418354    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418357    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418360    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418363    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418366    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418368    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418371    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418374    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418376    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418379    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418382    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418385    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418388    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418390    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418393    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:10:37.572389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418395    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418398    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418400    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418403    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418405    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418408    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418410    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418413    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418416    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418419    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418421    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418424    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.418427    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419096    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419102    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419106    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419108    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419111    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419114    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419116    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:10:37.586527 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419119    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419121    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419124    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419127    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419130    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419133    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419136    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419138    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419141    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419143    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419146    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419149    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419151    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419154    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419156    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419159    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419161    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419164    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419166    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419169    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:10:37.589082 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419173    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419176    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419178    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419181    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419184    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419187    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419190    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419192    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419195    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419197    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419200    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419202    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419205    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419208    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419210    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419213    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419216    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419218    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419221    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:10:37.589879 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419223    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419226    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419229    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419231    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419234    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419236    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419239    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419241    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419244    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419249    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419252    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419255    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419259    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419264    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419268    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419271    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419274    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419277    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419279    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:10:37.592773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419282    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419285    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419287    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419290    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419293    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419296    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419298    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419301    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419303    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419306    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419309    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419326    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419329    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419332    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419334    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419337    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419339    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419342    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419345    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419348    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:10:37.594041 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419350    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419353    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419355    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419358    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419360    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419363    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419367    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419371    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419374    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419378    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419381    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419383    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419386    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419389    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419730    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419734    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419737    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419740    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419743    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:10:37.595077 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419745    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419748    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419750    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419753    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419756    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419758    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419762    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419765    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419767    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419770    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419773    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419775    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419778    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419780    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419783    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419785    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419788    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419790    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419792    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419796    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:10:37.595862 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419799    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419801    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419804    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419806    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419809    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419811    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419814    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419817    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419819    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419822    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419824    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419828    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419831    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419835    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419838    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419841    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419845    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419848    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419850    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419853    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:10:37.596678 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419856    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419859    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419862    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419865    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419867    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419870    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419873    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419876    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419878    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419881    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419884    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419887    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419890    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419893    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419896    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419899    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419901    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419904    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419906    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:10:37.597281 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419910    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419913    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419916    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419919    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419923    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419926    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419929    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419932    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419934    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419937    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419940    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419942    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419945    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419947    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419950    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419952    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419954    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419957    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419959    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:10:37.597880 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419962    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419964    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419967    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419969    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419972    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419975    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419978    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419980    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419983    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419986    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419989    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419993    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419996    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.419999    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420001    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420004    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420086    2574 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420098    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420108    2574 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420114    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420118    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 00:10:37.598461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420121    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420125    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420129    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420133    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420136    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420139    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420142    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420146    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420149    2574 flags.go:64] FLAG: --cgroup-root=""
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420152    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420155    2574 flags.go:64] FLAG: --client-ca-file=""
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420158    2574 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420161    2574 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420165    2574 flags.go:64] FLAG: --cluster-domain=""
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420167    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420170    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420175    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420178    2574 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420182    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420185    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420188    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420192    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420195    2574 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420198    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 00:10:37.599083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420201    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420204    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420207    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420210    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420215    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420218    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420221    2574 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420223    2574 flags.go:64] FLAG: --enable-server="true"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420226    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420231    2574 flags.go:64] FLAG: --event-burst="100"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420234    2574 flags.go:64] FLAG: --event-qps="50"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420237    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420241    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420243    2574 flags.go:64] FLAG: --eviction-hard=""
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420247    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420250    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420253    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420256    2574 flags.go:64] FLAG: --eviction-soft=""
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420259    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420261    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420264    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420267    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420270    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420273    2574 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420277    2574 flags.go:64] FLAG: --feature-gates=""
Jul 24 00:10:37.599706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420280    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420283    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420286    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420289    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420292    2574 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420295    2574 flags.go:64] FLAG: --help="false"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420298    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-139-9.ec2.internal"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420301    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420304    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420307    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420324    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420330    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420333    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420335    2574 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420338    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420341    2574 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420344    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420347    2574 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420350    2574 flags.go:64] FLAG: --kube-reserved=""
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420353    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420356    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420359    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420362    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420365    2574 flags.go:64] FLAG: --lock-file=""
Jul 24 00:10:37.600450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420368    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420371    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420374    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420379    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420382    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420385    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420388    2574 flags.go:64] FLAG: --logging-format="text"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420392    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420395    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420398    2574 flags.go:64] FLAG: --manifest-url=""
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420401    2574 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420405    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420408    2574 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420411    2574 flags.go:64] FLAG: --max-pods="110"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420415    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420418    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420420    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420424    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420427    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420432    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420435    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420450    2574 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420453    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420456    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 00:10:37.601395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420459    2574 flags.go:64] FLAG: --pod-cidr=""
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420462    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420468    2574 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420470    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420473    2574 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420477    2574 flags.go:64] FLAG: --port="10250"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420480    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420482    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e465fe6e9570edaf"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420486    2574 flags.go:64] FLAG: --qos-reserved=""
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420488    2574 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420491    2574 flags.go:64] FLAG: --register-node="true"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420494    2574 flags.go:64] FLAG: --register-with-taints=""
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420503    2574 flags.go:64] FLAG: --registry-burst="10"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420505    2574 flags.go:64] FLAG: --registry-qps="5"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420508    2574 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420511    2574 flags.go:64] FLAG: --reserved-memory=""
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420516    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420519    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420522    2574 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420525    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420528    2574 flags.go:64] FLAG: --runonce="false"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420531    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420534    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420537    2574 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420539    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 00:10:37.602055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420542    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420545    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420550    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420552    2574 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420555    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420558    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420561    2574 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420564    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420567    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420570    2574 flags.go:64] FLAG: --system-cgroups=""
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420572    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420578    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420580    2574 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420584    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420588    2574 flags.go:64] FLAG: --tls-min-version=""
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420591    2574 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420593    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420596    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420599    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420602    2574 flags.go:64] FLAG: --v="2"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420605    2574 flags.go:64] FLAG: --version="false"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420609    2574 flags.go:64] FLAG: --vmodule=""
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420613    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.420617    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420728    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:10:37.767720 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420732    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:10:37.621056 ip-10-0-139-9 systemd[1]: Started Kubernetes Kubelet.
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420735    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420738    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420741    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420743    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420746    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420749    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420751    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420755    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420761    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420764    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420767    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420770    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420773    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420776    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420778    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420781    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420784    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420786    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:10:37.780877 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420789    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420791    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420794    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420797    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420800    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420802    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420805    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420808    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420810    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420813    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420815    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420819    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420822    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420824    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420828    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420832    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420835    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420837    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420840    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420843    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:10:37.781863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420845    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420848    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420852    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420854    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420857    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420860    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420862    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420865    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420867    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420870    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420874    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420877    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420881    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420883    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420886    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420890    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420892    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420895    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420898    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:10:37.782674 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420900    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420903    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420905    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420908    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420912    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420915    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420918    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420920    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420923    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420925    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420928    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420930    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420933    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420936    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420938    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420942    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420945    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420947    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420950    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420953    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:10:37.783277 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420955    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420958    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420960    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420963    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420966    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420969    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420971    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420974    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420976    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420979    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420981    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420984    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420986    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420989    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420992    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420994    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.420998    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.421001    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.421003    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:10:37.784083 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.421006    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.421012    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.429297    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.429343    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.429369    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.429376    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429450    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429457    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429462    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429468    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429472    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429477    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429481    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429484    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429488    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:10:37.784706 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429492    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429496    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429500    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429503    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429508    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429514    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429519    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429524    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429528    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429532    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429536    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429540    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429546    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429550    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429555    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429560    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429564    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429568    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429573    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:10:37.785273 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429578    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429582    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429590    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429594    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429598    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429602    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429606    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429609    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429613    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429618    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429621    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429625    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429628    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429632    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429637    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429640    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429644    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429648    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429652    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429657    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:10:37.785853 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429660    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429664    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429668    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429672    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429675    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429679    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429684    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429688    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429692    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429698    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429703    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429707    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429710    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429714    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429720    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429724    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429732    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429736    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429740    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:10:37.786510 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429744    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429748    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429752    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429756    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429759    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429763    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429767    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429771    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429775    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429778    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429783    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429787    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429791    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429795    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429799    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429803    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429807    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429810    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429814    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429818    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:10:37.787069 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429822    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429825    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429829    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429832    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429837    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429841    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429844    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429849    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429854    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429859    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429863    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.429867    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.429874    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430045    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430053    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:10:37.787708 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430057    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430063    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430069    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430073    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430077    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430082    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430086    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430090    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430094    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430098    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430102    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430106    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430110    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430114    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430117    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430121    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430125    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430129    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430133    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430137    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:10:37.788168 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430141    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430145    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430148    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430152    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430157    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430160    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430164    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430169    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430173    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430177    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430180    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430184    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430188    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430192    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430196    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430199    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430204    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430208    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430211    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:10:37.788786 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430215    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430219    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430224    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430227    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430232    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430236    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430240    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430244    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430248    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430251    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430255    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430259    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430262    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430266    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430270    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430274    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430278    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430284    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430287    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:10:37.789342 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430291    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430297    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430300    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430304    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430308    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430328    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430333    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430338    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430342    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430346    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430350    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430354    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430358    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430362    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430365    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430369    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430373    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430377    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430385    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430390    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:10:37.789896 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430393    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430397    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430400    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430404    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430410    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430414    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430418    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430423    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430427    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430431    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430437    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430442    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430447    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430453    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430457    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430461    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430465    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430469    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:10:37.790448 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:37.430473    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.430480    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.430610    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.561282    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.562739    2574 server.go:1019] "Starting client certificate rotation"
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.562869    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.562920    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.565749    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.567656    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.581159    2574 log.go:25] "Validated CRI v1 runtime API"
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.584427    2574 log.go:25] "Validated CRI v1 image API"
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.587536    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 00:10:37.790989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.592574    2574 fs.go:135] Filesystem UUIDs: map[1e218797-911e-4a3b-a591-adf9aa772a02:/dev/nvme0n1p3 59a8b18e-9103-4611-bbea-fbccfbad3676:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 24 00:10:37.791449 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.592597    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 00:10:37.791449 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.598773    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.603200    2574 manager.go:216] Machine: {Timestamp:2026-07-24 00:10:37.60153144 +0000 UTC m=+0.449850178 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100574 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec282c2446f5b647cc03a0e139a6c053 SystemUUID:ec282c24-46f5-b647-cc03-a0e139a6c053 BootID:04b5786d-938f-4e83-b928-0506b14aa577 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b2:f4:8d:30:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b2:f4:8d:30:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:6f:07:55:b8:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.603338    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.603411    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.604293    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.604332    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-9.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.605448    2574 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.605459    2574 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.605466    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.605481    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.606132    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.607930    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.608042    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.608070    2574 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.608080    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.608094    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.608103    2574 kubelet.go:398] "Adding apiserver pod source"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.608112    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 00:10:37.792084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.608721    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.609761    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.613100    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.614789    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.614817    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616283    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616297    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616303    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616308    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616336    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616345    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616354    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616359    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616365    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616370    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616379    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.616385    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.617111    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.617120    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.618332    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7pl6w"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.618639    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.618776    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-9.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.618824    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.620157    2574 server.go:1268] "Started kubelet"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.620268    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.621108    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.621202    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.621447    2574 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.622504    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.624086    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7pl6w"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.626460    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.630099    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.630102    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.630206    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-9.ec2.internal\" not found"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.630733    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.630817    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.630952    2574 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.630959    2574 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.631137    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.631929    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.631940    2574 factory.go:55] Registering systemd factory
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.631947    2574 factory.go:223] Registration of the systemd container factory successfully
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.632258    2574 factory.go:153] Registering CRI-O factory
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.632274    2574 factory.go:223] Registration of the crio container factory successfully
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.632297    2574 factory.go:103] Registering Raw factory
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.632331    2574 manager.go:1196] Started watching for new ooms in manager
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.632813    2574 manager.go:349] Starting recovery of all containers
Jul 24 00:10:37.792707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.636008    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.639558    2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-9.ec2.internal\" not found" node="ip-10-0-139-9.ec2.internal"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.647741    2574 manager.go:354] Recovery completed
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.652395    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.654054    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.654080    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.654091    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.654602    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.654613    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.654628    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.654608    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.655379    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.655396    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.655410    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.656086    2574 policy_none.go:49] "None policy: Start"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.656098    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.656108    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.656931    2574 policy_none.go:47] "Start"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.689521    2574 manager.go:337] "Starting Device Plugin manager"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.689652    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.689661    2574 server.go:87] "Starting device plugin registration server"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.689843    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.689851    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.691446    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.691498    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.691503    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.691940    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.691979    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-9.ec2.internal\" not found"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.744962    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.746029    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.746062    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.746083    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.746113    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.748672    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.790543    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.791261    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.791286    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.791327    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.791378    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-9.ec2.internal"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.800833    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-9.ec2.internal"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.800851    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-9.ec2.internal\": node \"ip-10-0-139-9.ec2.internal\" not found"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.816626    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-9.ec2.internal\" not found"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.846298    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal"]
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.846380    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.847183    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.847215    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.847230    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:10:37.916555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.848393    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.848535    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.848558    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.849411    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.849438    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.849461    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.851454    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.851490    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.851505    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.852542    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.852570    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.853238    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.853286    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.853301    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:10:37.918106 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:37.916661    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-9.ec2.internal\" not found"
Jul 24 00:10:38.026766 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.932522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68d9b39a00920ba6f2d51237ba0e930b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal\" (UID: \"68d9b39a00920ba6f2d51237ba0e930b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:38.026766 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.932543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68d9b39a00920ba6f2d51237ba0e930b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal\" (UID: \"68d9b39a00920ba6f2d51237ba0e930b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:38.026766 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:37.932559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf7019e1593e3ce75a451d35149a13f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-9.ec2.internal\" (UID: \"adf7019e1593e3ce75a451d35149a13f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:38.026766 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.017344    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-9.ec2.internal\" not found"
Jul 24 00:10:38.037561 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.033221    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68d9b39a00920ba6f2d51237ba0e930b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal\" (UID: \"68d9b39a00920ba6f2d51237ba0e930b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:38.037561 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.033245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf7019e1593e3ce75a451d35149a13f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-9.ec2.internal\" (UID: \"adf7019e1593e3ce75a451d35149a13f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:38.037561 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.033271    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68d9b39a00920ba6f2d51237ba0e930b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal\" (UID: \"68d9b39a00920ba6f2d51237ba0e930b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:38.037561 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.033329    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68d9b39a00920ba6f2d51237ba0e930b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal\" (UID: \"68d9b39a00920ba6f2d51237ba0e930b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:38.037561 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.033337    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf7019e1593e3ce75a451d35149a13f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-9.ec2.internal\" (UID: \"adf7019e1593e3ce75a451d35149a13f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:38.037561 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.033346    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68d9b39a00920ba6f2d51237ba0e930b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal\" (UID: \"68d9b39a00920ba6f2d51237ba0e930b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:38.134719 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.118164    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-9.ec2.internal\" not found"
Jul 24 00:10:38.244733 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.218397    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-9.ec2.internal\" not found"
Jul 24 00:10:38.357473 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.318763    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-9.ec2.internal\" not found"
Jul 24 00:10:38.465090 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.419009    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-9.ec2.internal\" not found"
Jul 24 00:10:38.570915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.499735    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:10:38.570915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.530509    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:38.570915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.541273    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 00:10:38.570915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.542840    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:38.570915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.550696    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 00:10:38.570915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.562758    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 00:10:38.570915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.562856    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 00:10:38.570915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.562861    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 00:10:38.570915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.562861    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.608924    2574 apiserver.go:52] "Watching apiserver"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.629140    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 00:05:37 +0000 UTC" deadline="2028-01-11 19:22:32.306933016 +0000 UTC"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.629182    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12883h11m53.677754149s"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.631243    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.644126    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.644437    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-vh7wc","openshift-multus/network-metrics-daemon-fmf4j","openshift-network-diagnostics/network-check-target-95fmv","openshift-network-operator/iptables-alerter-8bw7w","kube-system/konnectivity-agent-p7v47","openshift-cluster-node-tuning-operator/tuned-7rcjj","openshift-dns/node-resolver-ksmwx","openshift-image-registry/node-ca-kdddd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal","openshift-multus/multus-hsbpp","openshift-ovn-kubernetes/ovnkube-node-dlsrd","kube-system/global-pull-secret-syncer-j26n7","kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"]
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.645690    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ksmwx"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.646606    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.647748    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.647808    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.648459    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.648643    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8bw7w"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.649038    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2r45s\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.649603    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.650635    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.651747    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.652246    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.652261    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.652409    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.652479    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.652491    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.653055    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kdddd"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.653419    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.653445    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.654170    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.654219    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.654485    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7l755\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.654475    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.655707    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.657603    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.658062    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5rq8f\""
Jul 24 00:10:38.682550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.658070    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dzd2v\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.658073    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.658063    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.658185    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.658872    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.659197    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.659238    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.663618    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.663625    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.663884    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.664080    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.664082    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.664131    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.664132    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zs9qf\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.664247    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.664351    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7znpr\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.664387    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.664436    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.665013    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.665022    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.665039    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.665159    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2jzxh\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.665174    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.665302    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.665363    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hcvp8\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.665372    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vxsd9\""
Jul 24 00:10:38.683695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.672434    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.705442    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xdlwq"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.718434    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xdlwq"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.732029    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-cnibin\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737147    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-systemd\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737161    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-var-lib-kubelet\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737179    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz2qc\" (UniqueName: \"kubernetes.io/projected/f5b7f667-f9ea-4deb-9cb8-362fe65e017e-kube-api-access-gz2qc\") pod \"node-ca-kdddd\" (UID: \"f5b7f667-f9ea-4deb-9cb8-362fe65e017e\") " pod="openshift-image-registry/node-ca-kdddd"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737210    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a256a9ba-91cf-47ba-8e3b-a987081eac63-dbus\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737232    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-run-multus-certs\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737246    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-sysconfig\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737262    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-run\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737277    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a256a9ba-91cf-47ba-8e3b-a987081eac63-kubelet-config\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737291    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-etc-openvswitch\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737321    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c46gl\" (UniqueName: \"kubernetes.io/projected/7151f197-94ad-49a6-966a-1016bf6e4e9a-kube-api-access-c46gl\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737336    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e12a6b24-c866-4f4d-b583-719700a08c6e-host-slash\") pod \"iptables-alerter-8bw7w\" (UID: \"e12a6b24-c866-4f4d-b583-719700a08c6e\") " pod="openshift-network-operator/iptables-alerter-8bw7w"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737349    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1-konnectivity-ca\") pod \"konnectivity-agent-p7v47\" (UID: \"6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1\") " pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-sys-fs\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.799513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737384    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-run-k8s-cni-cncf-io\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737408    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-run-netns\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737431    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72dgc\" (UniqueName: \"kubernetes.io/projected/6364a346-b501-4735-a5c3-97ee5b0120af-kube-api-access-72dgc\") pod \"node-resolver-ksmwx\" (UID: \"6364a346-b501-4735-a5c3-97ee5b0120af\") " pod="openshift-dns/node-resolver-ksmwx"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737448    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-registration-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737462    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw2ts\" (UniqueName: \"kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts\") pod \"network-check-target-95fmv\" (UID: \"b3e62089-0435-442b-b800-fb8a90646c90\") " pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7151f197-94ad-49a6-966a-1016bf6e4e9a-ovnkube-config\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737506    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-daemon-config\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6364a346-b501-4735-a5c3-97ee5b0120af-tmp-dir\") pod \"node-resolver-ksmwx\" (UID: \"6364a346-b501-4735-a5c3-97ee5b0120af\") " pod="openshift-dns/node-resolver-ksmwx"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-log-socket\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737561    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737574    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-var-lib-cni-bin\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737587    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-cnibin\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737612    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5b7f667-f9ea-4deb-9cb8-362fe65e017e-host\") pod \"node-ca-kdddd\" (UID: \"f5b7f667-f9ea-4deb-9cb8-362fe65e017e\") " pod="openshift-image-registry/node-ca-kdddd"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-run-systemd\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737664    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-system-cni-dir\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.801036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737679    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acd4b7a0-11ee-40fb-ba36-9fb664430562-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-sys\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7151f197-94ad-49a6-966a-1016bf6e4e9a-env-overrides\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7151f197-94ad-49a6-966a-1016bf6e4e9a-ovnkube-script-lib\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737732    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhsz\" (UniqueName: \"kubernetes.io/projected/658582dc-8ffe-44c7-8866-ef8396278fb9-kube-api-access-mnhsz\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737745    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-tuned\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737798    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-kubelet\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737831    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-run-netns\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-device-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-cni-dir\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737892    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-etc-kubernetes\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737912    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-os-release\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737933    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-systemd-units\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737953    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-cni-bin\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.801801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.737995    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhlc\" (UniqueName: \"kubernetes.io/projected/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-kube-api-access-9zhlc\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738021    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-var-lib-kubelet\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-conf-dir\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-cni-netd\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738083    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-socket-dir-parent\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738097    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acd4b7a0-11ee-40fb-ba36-9fb664430562-cni-binary-copy\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6n6\" (UniqueName: \"kubernetes.io/projected/acd4b7a0-11ee-40fb-ba36-9fb664430562-kube-api-access-sb6n6\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738150    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6364a346-b501-4735-a5c3-97ee5b0120af-hosts-file\") pod \"node-resolver-ksmwx\" (UID: \"6364a346-b501-4735-a5c3-97ee5b0120af\") " pod="openshift-dns/node-resolver-ksmwx"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738170    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-kubernetes\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738183    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-slash\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738195    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-run-openvswitch\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7151f197-94ad-49a6-966a-1016bf6e4e9a-ovn-node-metrics-cert\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738236    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e12a6b24-c866-4f4d-b583-719700a08c6e-iptables-alerter-script\") pod \"iptables-alerter-8bw7w\" (UID: \"e12a6b24-c866-4f4d-b583-719700a08c6e\") " pod="openshift-network-operator/iptables-alerter-8bw7w"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1-agent-certs\") pod \"konnectivity-agent-p7v47\" (UID: \"6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1\") " pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738270    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-var-lib-cni-multus\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.802574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738283    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-host\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bss7n\" (UniqueName: \"kubernetes.io/projected/7558b14e-9a5e-481d-bcea-ba84760835dd-kube-api-access-bss7n\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-system-cni-dir\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738348    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-modprobe-d\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-var-lib-openvswitch\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738374    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-socket-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-os-release\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-lib-modules\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738453    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558b14e-9a5e-481d-bcea-ba84760835dd-tmp\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-run-ovn\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738499    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-node-log\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738545    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tksf7\" (UniqueName: \"kubernetes.io/projected/e12a6b24-c866-4f4d-b583-719700a08c6e-kube-api-access-tksf7\") pod \"iptables-alerter-8bw7w\" (UID: \"e12a6b24-c866-4f4d-b583-719700a08c6e\") " pod="openshift-network-operator/iptables-alerter-8bw7w"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738596    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/658582dc-8ffe-44c7-8866-ef8396278fb9-cni-binary-copy\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738616    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-sysctl-d\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.803353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5b7f667-f9ea-4deb-9cb8-362fe65e017e-serviceca\") pod \"node-ca-kdddd\" (UID: \"f5b7f667-f9ea-4deb-9cb8-362fe65e017e\") " pod="openshift-image-registry/node-ca-kdddd"
Jul 24 00:10:38.806077 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738665    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdq92\" (UniqueName: \"kubernetes.io/projected/10e19fc9-c926-4ecf-8748-2eff7762a0ff-kube-api-access-sdq92\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.806077 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-hostroot\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.806077 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acd4b7a0-11ee-40fb-ba36-9fb664430562-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.806077 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.738732    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-sysctl-conf\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.838844    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-var-lib-openvswitch\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.838865    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-socket-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.838889    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-os-release\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-lib-modules\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558b14e-9a5e-481d-bcea-ba84760835dd-tmp\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839187    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-run-ovn\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839190    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-var-lib-openvswitch\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839219    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-node-log\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839281    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-node-log\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839299    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839356    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-run-ovn\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839405    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839452    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839502    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-lib-modules\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tksf7\" (UniqueName: \"kubernetes.io/projected/e12a6b24-c866-4f4d-b583-719700a08c6e-kube-api-access-tksf7\") pod \"iptables-alerter-8bw7w\" (UID: \"e12a6b24-c866-4f4d-b583-719700a08c6e\") " pod="openshift-network-operator/iptables-alerter-8bw7w"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/658582dc-8ffe-44c7-8866-ef8396278fb9-cni-binary-copy\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.920691 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839611    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839567    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-os-release\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839637    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-sysctl-d\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839671    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5b7f667-f9ea-4deb-9cb8-362fe65e017e-serviceca\") pod \"node-ca-kdddd\" (UID: \"f5b7f667-f9ea-4deb-9cb8-362fe65e017e\") " pod="openshift-image-registry/node-ca-kdddd"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdq92\" (UniqueName: \"kubernetes.io/projected/10e19fc9-c926-4ecf-8748-2eff7762a0ff-kube-api-access-sdq92\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839733    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-hostroot\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839763    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acd4b7a0-11ee-40fb-ba36-9fb664430562-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-sysctl-conf\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839886    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-socket-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-cnibin\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.839985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-systemd\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-var-lib-kubelet\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2qc\" (UniqueName: \"kubernetes.io/projected/f5b7f667-f9ea-4deb-9cb8-362fe65e017e-kube-api-access-gz2qc\") pod \"node-ca-kdddd\" (UID: \"f5b7f667-f9ea-4deb-9cb8-362fe65e017e\") " pod="openshift-image-registry/node-ca-kdddd"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840066    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a256a9ba-91cf-47ba-8e3b-a987081eac63-dbus\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840098    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-run-multus-certs\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840104    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-sysctl-d\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840124    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-sysconfig\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.921461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840181    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-var-lib-kubelet\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840184    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-sysconfig\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840220    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-run\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a256a9ba-91cf-47ba-8e3b-a987081eac63-kubelet-config\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840389    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-systemd\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840396    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-etc-openvswitch\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840459    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c46gl\" (UniqueName: \"kubernetes.io/projected/7151f197-94ad-49a6-966a-1016bf6e4e9a-kube-api-access-c46gl\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840497    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e12a6b24-c866-4f4d-b583-719700a08c6e-host-slash\") pod \"iptables-alerter-8bw7w\" (UID: \"e12a6b24-c866-4f4d-b583-719700a08c6e\") " pod="openshift-network-operator/iptables-alerter-8bw7w"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840508    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-run-multus-certs\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840532    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-cnibin\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840581    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-sysctl-conf\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840599    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e12a6b24-c866-4f4d-b583-719700a08c6e-host-slash\") pod \"iptables-alerter-8bw7w\" (UID: \"e12a6b24-c866-4f4d-b583-719700a08c6e\") " pod="openshift-network-operator/iptables-alerter-8bw7w"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840655    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-hostroot\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840768    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-etc-openvswitch\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840773    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a256a9ba-91cf-47ba-8e3b-a987081eac63-dbus\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840807    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1-konnectivity-ca\") pod \"konnectivity-agent-p7v47\" (UID: \"6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1\") " pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840853    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-sys-fs\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840944    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-run\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.924161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-run-k8s-cni-cncf-io\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840972    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a256a9ba-91cf-47ba-8e3b-a987081eac63-kubelet-config\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.840998    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-run-k8s-cni-cncf-io\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841021    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-sys-fs\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841046    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-run-netns\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841066    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72dgc\" (UniqueName: \"kubernetes.io/projected/6364a346-b501-4735-a5c3-97ee5b0120af-kube-api-access-72dgc\") pod \"node-resolver-ksmwx\" (UID: \"6364a346-b501-4735-a5c3-97ee5b0120af\") " pod="openshift-dns/node-resolver-ksmwx"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841092    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-registration-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2ts\" (UniqueName: \"kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts\") pod \"network-check-target-95fmv\" (UID: \"b3e62089-0435-442b-b800-fb8a90646c90\") " pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841132    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7151f197-94ad-49a6-966a-1016bf6e4e9a-ovnkube-config\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841150    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-run-netns\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841177    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-daemon-config\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6364a346-b501-4735-a5c3-97ee5b0120af-tmp-dir\") pod \"node-resolver-ksmwx\" (UID: \"6364a346-b501-4735-a5c3-97ee5b0120af\") " pod="openshift-dns/node-resolver-ksmwx"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-log-socket\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841325    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841352    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-var-lib-cni-bin\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841400    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-cnibin\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841457    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5b7f667-f9ea-4deb-9cb8-362fe65e017e-host\") pod \"node-ca-kdddd\" (UID: \"f5b7f667-f9ea-4deb-9cb8-362fe65e017e\") " pod="openshift-image-registry/node-ca-kdddd"
Jul 24 00:10:38.924990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-run-systemd\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841547    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-var-lib-cni-bin\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841553    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841590    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-log-socket\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841604    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-system-cni-dir\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841634    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-run-systemd\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acd4b7a0-11ee-40fb-ba36-9fb664430562-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841648    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-cnibin\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-sys\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841697    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5b7f667-f9ea-4deb-9cb8-362fe65e017e-host\") pod \"node-ca-kdddd\" (UID: \"f5b7f667-f9ea-4deb-9cb8-362fe65e017e\") " pod="openshift-image-registry/node-ca-kdddd"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841711    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7151f197-94ad-49a6-966a-1016bf6e4e9a-env-overrides\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841742    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7151f197-94ad-49a6-966a-1016bf6e4e9a-ovnkube-script-lib\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841857    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-sys\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.841811    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhsz\" (UniqueName: \"kubernetes.io/projected/658582dc-8ffe-44c7-8866-ef8396278fb9-kube-api-access-mnhsz\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842520    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-tuned\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-kubelet\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.928212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-run-netns\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-device-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842619    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-cni-dir\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-etc-kubernetes\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-os-release\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842671    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-systemd-units\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842710    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-cni-bin\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842751    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhlc\" (UniqueName: \"kubernetes.io/projected/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-kube-api-access-9zhlc\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-var-lib-kubelet\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842871    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-conf-dir\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-cni-netd\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.842984    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-socket-dir-parent\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acd4b7a0-11ee-40fb-ba36-9fb664430562-cni-binary-copy\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6n6\" (UniqueName: \"kubernetes.io/projected/acd4b7a0-11ee-40fb-ba36-9fb664430562-kube-api-access-sb6n6\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843042    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6364a346-b501-4735-a5c3-97ee5b0120af-hosts-file\") pod \"node-resolver-ksmwx\" (UID: \"6364a346-b501-4735-a5c3-97ee5b0120af\") " pod="openshift-dns/node-resolver-ksmwx"
Jul 24 00:10:38.928988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843067    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-kubernetes\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843092    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-slash\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843139    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-run-openvswitch\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7151f197-94ad-49a6-966a-1016bf6e4e9a-ovn-node-metrics-cert\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843275    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e12a6b24-c866-4f4d-b583-719700a08c6e-iptables-alerter-script\") pod \"iptables-alerter-8bw7w\" (UID: \"e12a6b24-c866-4f4d-b583-719700a08c6e\") " pod="openshift-network-operator/iptables-alerter-8bw7w"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843293    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843395    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-run-netns\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843469    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-system-cni-dir\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843514    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-os-release\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843506    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1-agent-certs\") pod \"konnectivity-agent-p7v47\" (UID: \"6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1\") " pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-var-lib-cni-multus\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.843584    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843663    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-socket-dir-parent\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.843690    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret podName:a256a9ba-91cf-47ba-8e3b-a987081eac63 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:39.343647214 +0000 UTC m=+2.191965951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret") pod "global-pull-secret-syncer-j26n7" (UID: "a256a9ba-91cf-47ba-8e3b-a987081eac63") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843699    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-cni-bin\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843802    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-run-openvswitch\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843795    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-var-lib-kubelet\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.931681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.843922    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-conf-dir\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-cni-netd\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844353    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-device-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844388    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-systemd-units\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.844415    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844438    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-host\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.844489    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs podName:1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:39.344470638 +0000 UTC m=+2.192789371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs") pod "network-metrics-daemon-fmf4j" (UID: "1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844508    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bss7n\" (UniqueName: \"kubernetes.io/projected/7558b14e-9a5e-481d-bcea-ba84760835dd-kube-api-access-bss7n\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844525    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-host\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844572    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-host-var-lib-cni-multus\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-system-cni-dir\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844621    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e19fc9-c926-4ecf-8748-2eff7762a0ff-registration-dir\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844625    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-modprobe-d\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844418    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-etc-kubernetes\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844693    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6364a346-b501-4735-a5c3-97ee5b0120af-hosts-file\") pod \"node-resolver-ksmwx\" (UID: \"6364a346-b501-4735-a5c3-97ee5b0120af\") " pod="openshift-dns/node-resolver-ksmwx"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844705    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844778    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-kubernetes\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.932459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.844912    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acd4b7a0-11ee-40fb-ba36-9fb664430562-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.845487    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.845519    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-kubelet\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.845636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-modprobe-d\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.845700    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-cni-dir\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.845719    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6364a346-b501-4735-a5c3-97ee5b0120af-tmp-dir\") pod \"node-resolver-ksmwx\" (UID: \"6364a346-b501-4735-a5c3-97ee5b0120af\") " pod="openshift-dns/node-resolver-ksmwx"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.845739    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/658582dc-8ffe-44c7-8866-ef8396278fb9-multus-daemon-config\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.845771    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acd4b7a0-11ee-40fb-ba36-9fb664430562-system-cni-dir\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.845820    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acd4b7a0-11ee-40fb-ba36-9fb664430562-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.846099    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7151f197-94ad-49a6-966a-1016bf6e4e9a-ovnkube-config\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.846180    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7151f197-94ad-49a6-966a-1016bf6e4e9a-env-overrides\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.846185    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7151f197-94ad-49a6-966a-1016bf6e4e9a-host-slash\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.846393    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1-konnectivity-ca\") pod \"konnectivity-agent-p7v47\" (UID: \"6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1\") " pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.846519    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7151f197-94ad-49a6-966a-1016bf6e4e9a-ovnkube-script-lib\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.846731    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acd4b7a0-11ee-40fb-ba36-9fb664430562-cni-binary-copy\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.846775    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e12a6b24-c866-4f4d-b583-719700a08c6e-iptables-alerter-script\") pod \"iptables-alerter-8bw7w\" (UID: \"e12a6b24-c866-4f4d-b583-719700a08c6e\") " pod="openshift-network-operator/iptables-alerter-8bw7w"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.846848    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/658582dc-8ffe-44c7-8866-ef8396278fb9-cni-binary-copy\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.847019    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5b7f667-f9ea-4deb-9cb8-362fe65e017e-serviceca\") pod \"node-ca-kdddd\" (UID: \"f5b7f667-f9ea-4deb-9cb8-362fe65e017e\") " pod="openshift-image-registry/node-ca-kdddd"
Jul 24 00:10:38.936292 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.848019    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.848038    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.848048    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zw2ts for pod openshift-network-diagnostics/network-check-target-95fmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:38.848093    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts podName:b3e62089-0435-442b-b800-fb8a90646c90 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:39.348080646 +0000 UTC m=+2.196399385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zw2ts" (UniqueName: "kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts") pod "network-check-target-95fmv" (UID: "b3e62089-0435-442b-b800-fb8a90646c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.848331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7558b14e-9a5e-481d-bcea-ba84760835dd-etc-tuned\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.848607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558b14e-9a5e-481d-bcea-ba84760835dd-tmp\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.848610    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7151f197-94ad-49a6-966a-1016bf6e4e9a-ovn-node-metrics-cert\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.849281    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1-agent-certs\") pod \"konnectivity-agent-p7v47\" (UID: \"6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1\") " pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.850438    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tksf7\" (UniqueName: \"kubernetes.io/projected/e12a6b24-c866-4f4d-b583-719700a08c6e-kube-api-access-tksf7\") pod \"iptables-alerter-8bw7w\" (UID: \"e12a6b24-c866-4f4d-b583-719700a08c6e\") " pod="openshift-network-operator/iptables-alerter-8bw7w"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.850473    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72dgc\" (UniqueName: \"kubernetes.io/projected/6364a346-b501-4735-a5c3-97ee5b0120af-kube-api-access-72dgc\") pod \"node-resolver-ksmwx\" (UID: \"6364a346-b501-4735-a5c3-97ee5b0120af\") " pod="openshift-dns/node-resolver-ksmwx"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.850502    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c46gl\" (UniqueName: \"kubernetes.io/projected/7151f197-94ad-49a6-966a-1016bf6e4e9a-kube-api-access-c46gl\") pod \"ovnkube-node-dlsrd\" (UID: \"7151f197-94ad-49a6-966a-1016bf6e4e9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.850839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2qc\" (UniqueName: \"kubernetes.io/projected/f5b7f667-f9ea-4deb-9cb8-362fe65e017e-kube-api-access-gz2qc\") pod \"node-ca-kdddd\" (UID: \"f5b7f667-f9ea-4deb-9cb8-362fe65e017e\") " pod="openshift-image-registry/node-ca-kdddd"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.851041    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhsz\" (UniqueName: \"kubernetes.io/projected/658582dc-8ffe-44c7-8866-ef8396278fb9-kube-api-access-mnhsz\") pod \"multus-hsbpp\" (UID: \"658582dc-8ffe-44c7-8866-ef8396278fb9\") " pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.851906    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdq92\" (UniqueName: \"kubernetes.io/projected/10e19fc9-c926-4ecf-8748-2eff7762a0ff-kube-api-access-sdq92\") pod \"aws-ebs-csi-driver-node-nfqj2\" (UID: \"10e19fc9-c926-4ecf-8748-2eff7762a0ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.851915    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhlc\" (UniqueName: \"kubernetes.io/projected/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-kube-api-access-9zhlc\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.852034    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6n6\" (UniqueName: \"kubernetes.io/projected/acd4b7a0-11ee-40fb-ba36-9fb664430562-kube-api-access-sb6n6\") pod \"multus-additional-cni-plugins-vh7wc\" (UID: \"acd4b7a0-11ee-40fb-ba36-9fb664430562\") " pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:38.937039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:38.852642    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bss7n\" (UniqueName: \"kubernetes.io/projected/7558b14e-9a5e-481d-bcea-ba84760835dd-kube-api-access-bss7n\") pod \"tuned-7rcjj\" (UID: \"7558b14e-9a5e-481d-bcea-ba84760835dd\") " pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:39.170771 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.115791    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:10:39.347396 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.347372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:39.347518 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.347408    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:39.347554 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:39.347539    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:39.347617 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:39.347606    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs podName:1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:40.347588699 +0000 UTC m=+3.195907432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs") pod "network-metrics-daemon-fmf4j" (UID: "1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:39.347659 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:39.347549    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:39.347691 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:39.347677    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret podName:a256a9ba-91cf-47ba-8e3b-a987081eac63 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:40.347659741 +0000 UTC m=+3.195978460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret") pod "global-pull-secret-syncer-j26n7" (UID: "a256a9ba-91cf-47ba-8e3b-a987081eac63") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:39.434934 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.434868    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:39.439957 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.439939    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:39.448188 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.448167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2ts\" (UniqueName: \"kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts\") pod \"network-check-target-95fmv\" (UID: \"b3e62089-0435-442b-b800-fb8a90646c90\") " pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:39.448285 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:39.448273    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:10:39.448333 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:39.448290    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:10:39.448333 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:39.448298    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zw2ts for pod openshift-network-diagnostics/network-check-target-95fmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:39.448408 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:39.448354    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts podName:b3e62089-0435-442b-b800-fb8a90646c90 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:40.448341375 +0000 UTC m=+3.296660094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zw2ts" (UniqueName: "kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts") pod "network-check-target-95fmv" (UID: "b3e62089-0435-442b-b800-fb8a90646c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:39.449213 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.449201    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 00:10:39.449247 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:39.449238    2574 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:39.449431 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.449420    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:39.449470 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.449425    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 00:10:39.449500 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:39.449491    2574 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-9.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:39.449649 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.449639    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal"
Jul 24 00:10:39.469359 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.469341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ksmwx"
Jul 24 00:10:39.484951 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.484927    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:10:39.504496 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.504480    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8bw7w"
Jul 24 00:10:39.517770 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.517754    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:10:39.523375 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.523357    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7rcjj"
Jul 24 00:10:39.541295 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.541277    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2"
Jul 24 00:10:39.561859 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.561835    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kdddd"
Jul 24 00:10:39.581992 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.581968    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hsbpp"
Jul 24 00:10:39.586890 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.586872    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vh7wc"
Jul 24 00:10:39.624903 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.624880    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:10:39.720463 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.720386    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 00:05:38 +0000 UTC" deadline="2028-03-16 14:41:07.297533502 +0000 UTC"
Jul 24 00:10:39.720463 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:39.720422    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14438h30m27.577113798s"
Jul 24 00:10:40.054603 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:40.054558    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf7019e1593e3ce75a451d35149a13f.slice/crio-3706381ea53b96ad6653ed2e9f866e85559df9aba3ebec5cf275c2231bfa0011 WatchSource:0}: Error finding container 3706381ea53b96ad6653ed2e9f866e85559df9aba3ebec5cf275c2231bfa0011: Status 404 returned error can't find the container with id 3706381ea53b96ad6653ed2e9f866e85559df9aba3ebec5cf275c2231bfa0011
Jul 24 00:10:40.055090 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:40.055067    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7151f197_94ad_49a6_966a_1016bf6e4e9a.slice/crio-f98cc9d0d9da92293497b74c0d35c55bf9cf8c7c250a247d78632858d712344f WatchSource:0}: Error finding container f98cc9d0d9da92293497b74c0d35c55bf9cf8c7c250a247d78632858d712344f: Status 404 returned error can't find the container with id f98cc9d0d9da92293497b74c0d35c55bf9cf8c7c250a247d78632858d712344f
Jul 24 00:10:40.055704 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:40.055685    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7558b14e_9a5e_481d_bcea_ba84760835dd.slice/crio-1a20e721c0ad545821b507e9163c4246210e5b6d14706f3914810cc4e917fa75 WatchSource:0}: Error finding container 1a20e721c0ad545821b507e9163c4246210e5b6d14706f3914810cc4e917fa75: Status 404 returned error can't find the container with id 1a20e721c0ad545821b507e9163c4246210e5b6d14706f3914810cc4e917fa75
Jul 24 00:10:40.059649 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.059631    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:10:40.084700 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:40.084679    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e19fc9_c926_4ecf_8748_2eff7762a0ff.slice/crio-5d247e4709f51109cf312cf9adbab0578485e7d2fee6a530705cd3a00c2de752 WatchSource:0}: Error finding container 5d247e4709f51109cf312cf9adbab0578485e7d2fee6a530705cd3a00c2de752: Status 404 returned error can't find the container with id 5d247e4709f51109cf312cf9adbab0578485e7d2fee6a530705cd3a00c2de752
Jul 24 00:10:40.107389 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:40.107367    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6220ffd0_9ea8_41c3_b1c0_fa00c590bfb1.slice/crio-b28084d8bc4f8c7fdcbd71677268745db40b0be96e46df55bd683a52b7d2a8d3 WatchSource:0}: Error finding container b28084d8bc4f8c7fdcbd71677268745db40b0be96e46df55bd683a52b7d2a8d3: Status 404 returned error can't find the container with id b28084d8bc4f8c7fdcbd71677268745db40b0be96e46df55bd683a52b7d2a8d3
Jul 24 00:10:40.132289 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:40.132268    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b7f667_f9ea_4deb_9cb8_362fe65e017e.slice/crio-17486375fec7e408125f3ce5b2fb5eacb0a0b23e125c91680e6798d045899e48 WatchSource:0}: Error finding container 17486375fec7e408125f3ce5b2fb5eacb0a0b23e125c91680e6798d045899e48: Status 404 returned error can't find the container with id 17486375fec7e408125f3ce5b2fb5eacb0a0b23e125c91680e6798d045899e48
Jul 24 00:10:40.152286 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:40.152267    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12a6b24_c866_4f4d_b583_719700a08c6e.slice/crio-9344b7cfd05d1f8e450068972f2ead6dfc8b400a5020b8c1b63eacb916b32fa0 WatchSource:0}: Error finding container 9344b7cfd05d1f8e450068972f2ead6dfc8b400a5020b8c1b63eacb916b32fa0: Status 404 returned error can't find the container with id 9344b7cfd05d1f8e450068972f2ead6dfc8b400a5020b8c1b63eacb916b32fa0
Jul 24 00:10:40.163180 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.163161    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:10:40.177571 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:40.177547    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658582dc_8ffe_44c7_8866_ef8396278fb9.slice/crio-59162192235dc538252e4a159fde4af7050dd8b60b91c040783dd46a3f756457 WatchSource:0}: Error finding container 59162192235dc538252e4a159fde4af7050dd8b60b91c040783dd46a3f756457: Status 404 returned error can't find the container with id 59162192235dc538252e4a159fde4af7050dd8b60b91c040783dd46a3f756457
Jul 24 00:10:40.228044 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:40.228023    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d9b39a00920ba6f2d51237ba0e930b.slice/crio-d42a33f4fdb1e0423666d9bde0baf04e88ea96a7935fb2e679254fd455fe954a WatchSource:0}: Error finding container d42a33f4fdb1e0423666d9bde0baf04e88ea96a7935fb2e679254fd455fe954a: Status 404 returned error can't find the container with id d42a33f4fdb1e0423666d9bde0baf04e88ea96a7935fb2e679254fd455fe954a
Jul 24 00:10:40.328563 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:10:40.328535    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6364a346_b501_4735_a5c3_97ee5b0120af.slice/crio-4b93d9e58a1764cb7923e5df7f50bf6264d604e91e0f95532b2e97accbe74201 WatchSource:0}: Error finding container 4b93d9e58a1764cb7923e5df7f50bf6264d604e91e0f95532b2e97accbe74201: Status 404 returned error can't find the container with id 4b93d9e58a1764cb7923e5df7f50bf6264d604e91e0f95532b2e97accbe74201
Jul 24 00:10:40.355060 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.355035    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:40.355143 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.355077    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:40.355185 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:40.355169    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:40.355218 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:40.355209    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret podName:a256a9ba-91cf-47ba-8e3b-a987081eac63 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:42.355197941 +0000 UTC m=+5.203516660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret") pod "global-pull-secret-syncer-j26n7" (UID: "a256a9ba-91cf-47ba-8e3b-a987081eac63") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:40.355258 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:40.355169    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:40.355258 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:40.355244    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs podName:1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:42.355237632 +0000 UTC m=+5.203556351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs") pod "network-metrics-daemon-fmf4j" (UID: "1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:40.456323 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.456288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2ts\" (UniqueName: \"kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts\") pod \"network-check-target-95fmv\" (UID: \"b3e62089-0435-442b-b800-fb8a90646c90\") " pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:40.456432 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:40.456418    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:10:40.456469 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:40.456437    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:10:40.456469 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:40.456445    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zw2ts for pod openshift-network-diagnostics/network-check-target-95fmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:40.456542 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:40.456491    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts podName:b3e62089-0435-442b-b800-fb8a90646c90 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:42.456472824 +0000 UTC m=+5.304791542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zw2ts" (UniqueName: "kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts") pod "network-check-target-95fmv" (UID: "b3e62089-0435-442b-b800-fb8a90646c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:40.721668 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.721554    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 00:05:38 +0000 UTC" deadline="2028-05-10 19:30:30.826594512 +0000 UTC"
Jul 24 00:10:40.721668 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.721595    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15763h19m50.10500417s"
Jul 24 00:10:40.747987 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.747168    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:40.747987 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:40.747292    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:10:40.747987 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.747709    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:40.747987 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:40.747813    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:10:40.747987 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.747876    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:40.747987 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:40.747946    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:10:40.760503 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.760427    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hsbpp" event={"ID":"658582dc-8ffe-44c7-8866-ef8396278fb9","Type":"ContainerStarted","Data":"59162192235dc538252e4a159fde4af7050dd8b60b91c040783dd46a3f756457"}
Jul 24 00:10:40.765687 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.765614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8bw7w" event={"ID":"e12a6b24-c866-4f4d-b583-719700a08c6e","Type":"ContainerStarted","Data":"9344b7cfd05d1f8e450068972f2ead6dfc8b400a5020b8c1b63eacb916b32fa0"}
Jul 24 00:10:40.769395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.769337    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kdddd" event={"ID":"f5b7f667-f9ea-4deb-9cb8-362fe65e017e","Type":"ContainerStarted","Data":"17486375fec7e408125f3ce5b2fb5eacb0a0b23e125c91680e6798d045899e48"}
Jul 24 00:10:40.772391 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.772273    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd" event={"ID":"7151f197-94ad-49a6-966a-1016bf6e4e9a","Type":"ContainerStarted","Data":"f98cc9d0d9da92293497b74c0d35c55bf9cf8c7c250a247d78632858d712344f"}
Jul 24 00:10:40.776534 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.776468    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7rcjj" event={"ID":"7558b14e-9a5e-481d-bcea-ba84760835dd","Type":"ContainerStarted","Data":"1a20e721c0ad545821b507e9163c4246210e5b6d14706f3914810cc4e917fa75"}
Jul 24 00:10:40.782153 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.782102    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vh7wc" event={"ID":"acd4b7a0-11ee-40fb-ba36-9fb664430562","Type":"ContainerStarted","Data":"2fe059d431fd4914f290c369185b018abea9ff3463419ebf86dc14ba76e2d9b1"}
Jul 24 00:10:40.790599 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.790570    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ksmwx" event={"ID":"6364a346-b501-4735-a5c3-97ee5b0120af","Type":"ContainerStarted","Data":"4b93d9e58a1764cb7923e5df7f50bf6264d604e91e0f95532b2e97accbe74201"}
Jul 24 00:10:40.797500 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.797451    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p7v47" event={"ID":"6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1","Type":"ContainerStarted","Data":"b28084d8bc4f8c7fdcbd71677268745db40b0be96e46df55bd683a52b7d2a8d3"}
Jul 24 00:10:40.799916 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.799821    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2" event={"ID":"10e19fc9-c926-4ecf-8748-2eff7762a0ff","Type":"ContainerStarted","Data":"5d247e4709f51109cf312cf9adbab0578485e7d2fee6a530705cd3a00c2de752"}
Jul 24 00:10:40.801457 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.801396    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal" event={"ID":"adf7019e1593e3ce75a451d35149a13f","Type":"ContainerStarted","Data":"3706381ea53b96ad6653ed2e9f866e85559df9aba3ebec5cf275c2231bfa0011"}
Jul 24 00:10:40.804205 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:40.804184    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal" event={"ID":"68d9b39a00920ba6f2d51237ba0e930b","Type":"ContainerStarted","Data":"d42a33f4fdb1e0423666d9bde0baf04e88ea96a7935fb2e679254fd455fe954a"}
Jul 24 00:10:41.654327 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:41.654281    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:10:42.372856 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:42.372819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:42.373414 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:42.372880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:42.373414 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:42.373081    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:42.373414 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:42.373148    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret podName:a256a9ba-91cf-47ba-8e3b-a987081eac63 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:46.373128604 +0000 UTC m=+9.221447338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret") pod "global-pull-secret-syncer-j26n7" (UID: "a256a9ba-91cf-47ba-8e3b-a987081eac63") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:42.373414 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:42.373270    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:42.373414 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:42.373363    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs podName:1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:46.373293969 +0000 UTC m=+9.221612692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs") pod "network-metrics-daemon-fmf4j" (UID: "1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:42.474199 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:42.474165    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2ts\" (UniqueName: \"kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts\") pod \"network-check-target-95fmv\" (UID: \"b3e62089-0435-442b-b800-fb8a90646c90\") " pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:42.474410 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:42.474392    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:10:42.474484 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:42.474421    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:10:42.474484 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:42.474434    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zw2ts for pod openshift-network-diagnostics/network-check-target-95fmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:42.474588 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:42.474516    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts podName:b3e62089-0435-442b-b800-fb8a90646c90 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:46.474496789 +0000 UTC m=+9.322815527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zw2ts" (UniqueName: "kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts") pod "network-check-target-95fmv" (UID: "b3e62089-0435-442b-b800-fb8a90646c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:42.748014 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:42.747128    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:42.748014 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:42.747268    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:10:42.748014 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:42.747683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:42.748014 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:42.747769    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:10:42.748014 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:42.747829    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:42.748014 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:42.747897    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:10:44.746928 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:44.746899    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:44.747443 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:44.747042    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:10:44.747443 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:44.747093    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:44.747443 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:44.747213    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:10:44.747443 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:44.747266    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:44.747443 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:44.747352    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:10:46.405964 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:46.405929    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:46.405964 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:46.405974    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:46.406512 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:46.406115    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:46.406512 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:46.406117    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:46.406512 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:46.406183    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret podName:a256a9ba-91cf-47ba-8e3b-a987081eac63 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:54.406169386 +0000 UTC m=+17.254488109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret") pod "global-pull-secret-syncer-j26n7" (UID: "a256a9ba-91cf-47ba-8e3b-a987081eac63") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:46.406512 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:46.406198    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs podName:1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:54.406191329 +0000 UTC m=+17.254510048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs") pod "network-metrics-daemon-fmf4j" (UID: "1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:46.506601 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:46.506562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2ts\" (UniqueName: \"kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts\") pod \"network-check-target-95fmv\" (UID: \"b3e62089-0435-442b-b800-fb8a90646c90\") " pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:46.506795 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:46.506734    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:10:46.506795 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:46.506760    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:10:46.506795 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:46.506774    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zw2ts for pod openshift-network-diagnostics/network-check-target-95fmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:46.506951 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:46.506836    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts podName:b3e62089-0435-442b-b800-fb8a90646c90 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:54.506821154 +0000 UTC m=+17.355139875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zw2ts" (UniqueName: "kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts") pod "network-check-target-95fmv" (UID: "b3e62089-0435-442b-b800-fb8a90646c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:46.746429 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:46.746357    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:46.746590 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:46.746362    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:46.746590 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:46.746488    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:10:46.746590 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:46.746566    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:10:46.746590 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:46.746365    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:46.746802 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:46.746645    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:10:48.746890 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:48.746683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:48.746890 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:48.746831    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:10:48.746890 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:48.746852    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:48.747530 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:48.746971    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:10:48.747530 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:48.747025    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:48.747530 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:48.747083    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:10:50.746457 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:50.746425    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:50.746891 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:50.746429    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:50.746891 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:50.746534    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:10:50.746891 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:50.746431    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:50.746891 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:50.746648    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:10:50.746891 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:50.746763    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:10:52.746828 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:52.746623    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:52.747275 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:52.746637    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:52.747275 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:52.746919    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:10:52.747275 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:52.746667    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:52.747275 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:52.747014    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:10:52.747275 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:52.747072    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:10:54.466488 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:54.466447    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:54.466944 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:54.466510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:54.466944 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:54.466573    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:54.466944 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:54.466606    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:54.466944 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:54.466659    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs podName:1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:10.466632584 +0000 UTC m=+33.314951317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs") pod "network-metrics-daemon-fmf4j" (UID: "1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:10:54.466944 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:54.466680    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret podName:a256a9ba-91cf-47ba-8e3b-a987081eac63 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:10.466669937 +0000 UTC m=+33.314988657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret") pod "global-pull-secret-syncer-j26n7" (UID: "a256a9ba-91cf-47ba-8e3b-a987081eac63") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:10:54.567143 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:54.567106    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2ts\" (UniqueName: \"kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts\") pod \"network-check-target-95fmv\" (UID: \"b3e62089-0435-442b-b800-fb8a90646c90\") " pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:54.567615 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:54.567566    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:10:54.567615 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:54.567610    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:10:54.567796 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:54.567624    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zw2ts for pod openshift-network-diagnostics/network-check-target-95fmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:54.567796 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:54.567694    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts podName:b3e62089-0435-442b-b800-fb8a90646c90 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:10.567673563 +0000 UTC m=+33.415992298 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zw2ts" (UniqueName: "kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts") pod "network-check-target-95fmv" (UID: "b3e62089-0435-442b-b800-fb8a90646c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:10:54.747366 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:54.747267    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:54.747366 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:54.747345    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:54.747581 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:54.747482    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:10:54.747581 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:54.747503    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:54.747689 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:54.747620    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:10:54.747737 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:54.747708    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:10:56.747327 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:56.747283    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:56.747749 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:56.747283    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:56.747749 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:56.747417    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:10:56.747749 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:56.747510    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:10:56.747749 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:56.747283    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:56.747749 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:56.747620    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:10:58.747076 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:58.747039    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:10:58.747076 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:58.747066    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:10:58.747589 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:58.747160    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:10:58.747589 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:10:58.747165    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:10:58.747589 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:58.747274    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:10:58.747589 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:10:58.747417    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:11:00.746710 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.746533    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:00.747235 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.746540    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:00.747235 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:00.746779    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:11:00.747235 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:00.746850    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:11:00.747235 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.746569    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:00.747235 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:00.747017    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:11:00.849025 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.848989    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p7v47" event={"ID":"6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1","Type":"ContainerStarted","Data":"dd0ce9c8cdea4d5a0c755c1d327d29babbbc07546ae51114c9f24f805e28cf8f"}
Jul 24 00:11:00.850631 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.850597    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2" event={"ID":"10e19fc9-c926-4ecf-8748-2eff7762a0ff","Type":"ContainerStarted","Data":"b48314914047627784d0edd67f328c3ed54499c54fd0f42dd7c3bb32446b836d"}
Jul 24 00:11:00.852019 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.851997    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal" event={"ID":"adf7019e1593e3ce75a451d35149a13f","Type":"ContainerStarted","Data":"8ecb290a3f49a1413db1e80c66c17a227356918dacc12a76f394a82bfc982bd1"}
Jul 24 00:11:00.853387 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.853364    2574 generic.go:356] "Generic (PLEG): container finished" podID="68d9b39a00920ba6f2d51237ba0e930b" containerID="6fcb71d92043b4551d73044ea6b5896f8256ab2e7dae5797679c66f42e512bf0" exitCode=0
Jul 24 00:11:00.853492 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.853434    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal" event={"ID":"68d9b39a00920ba6f2d51237ba0e930b","Type":"ContainerDied","Data":"6fcb71d92043b4551d73044ea6b5896f8256ab2e7dae5797679c66f42e512bf0"}
Jul 24 00:11:00.854806 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.854789    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hsbpp" event={"ID":"658582dc-8ffe-44c7-8866-ef8396278fb9","Type":"ContainerStarted","Data":"a67d16e9bd88e88b7186676e73ac5575b4b8def48cedb3a78a292086536254ca"}
Jul 24 00:11:00.856141 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.856122    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kdddd" event={"ID":"f5b7f667-f9ea-4deb-9cb8-362fe65e017e","Type":"ContainerStarted","Data":"9a8cee4ee48cb9268b53973930fc314284c485cdaf709b2af995739efd5b9ca4"}
Jul 24 00:11:00.858973 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.858946    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd" event={"ID":"7151f197-94ad-49a6-966a-1016bf6e4e9a","Type":"ContainerStarted","Data":"92b7e31b897d600c9ec975784558d50cad4b44de9b39b59504c1411fdaad95c5"}
Jul 24 00:11:00.859040 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.859016    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd" event={"ID":"7151f197-94ad-49a6-966a-1016bf6e4e9a","Type":"ContainerStarted","Data":"f872cbc29a65bfb790d7184e7a45b948128e791aea939225448678fa59a7bd1a"}
Jul 24 00:11:00.859093 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.859064    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd" event={"ID":"7151f197-94ad-49a6-966a-1016bf6e4e9a","Type":"ContainerStarted","Data":"ff40ed0a0e99fd440326a926902e8faf9e253f0ac616d694ca3925218ffb4eb5"}
Jul 24 00:11:00.859144 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.859103    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd" event={"ID":"7151f197-94ad-49a6-966a-1016bf6e4e9a","Type":"ContainerStarted","Data":"a38700ba46b38e99939468b474c74074c4f7bbe1e7dba4cb5fd1df9e58db7666"}
Jul 24 00:11:00.859144 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.859139    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd" event={"ID":"7151f197-94ad-49a6-966a-1016bf6e4e9a","Type":"ContainerStarted","Data":"a924cbf6b28b7ea47ffa5963dbe04f34e1b2ebc0dcb76aea9064e77df24b9cbd"}
Jul 24 00:11:00.859231 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.859176    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd" event={"ID":"7151f197-94ad-49a6-966a-1016bf6e4e9a","Type":"ContainerStarted","Data":"cd64abe8149b367191990445effd6800e082d301f25f32b684431d05a0f4c1d4"}
Jul 24 00:11:00.860238 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.860220    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7rcjj" event={"ID":"7558b14e-9a5e-481d-bcea-ba84760835dd","Type":"ContainerStarted","Data":"c2b00935798714f18f9529a3ecf4af5749c3b9b13ba7d3707e697d829a415bef"}
Jul 24 00:11:00.861581 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.861562    2574 generic.go:356] "Generic (PLEG): container finished" podID="acd4b7a0-11ee-40fb-ba36-9fb664430562" containerID="fd0f3977f2547abc61858de2451e7d3eca3f2f435ca7459c40d03518209c9444" exitCode=0
Jul 24 00:11:00.861665 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.861619    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vh7wc" event={"ID":"acd4b7a0-11ee-40fb-ba36-9fb664430562","Type":"ContainerDied","Data":"fd0f3977f2547abc61858de2451e7d3eca3f2f435ca7459c40d03518209c9444"}
Jul 24 00:11:00.863027 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.862995    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ksmwx" event={"ID":"6364a346-b501-4735-a5c3-97ee5b0120af","Type":"ContainerStarted","Data":"6401e70aedadaee79021d2a61f01d51ab910844ed073390f18a260d661bf9377"}
Jul 24 00:11:00.866070 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.866036    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p7v47" podStartSLOduration=4.197678497 podStartE2EDuration="23.866025647s" totalImagesPullingTime="19.66834715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:00.865834225 +0000 UTC m=+23.714152965" watchObservedRunningTime="2026-07-24 00:11:00.866025647 +0000 UTC m=+23.714344388"
Jul 24 00:11:00.896612 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.896390    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kdddd" podStartSLOduration=4.241000661 podStartE2EDuration="23.896370878s" totalImagesPullingTime="19.655370217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:00.895980371 +0000 UTC m=+23.744299112" watchObservedRunningTime="2026-07-24 00:11:00.896370878 +0000 UTC m=+23.744689623"
Jul 24 00:11:00.954238 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.954196    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ksmwx" podStartSLOduration=4.506980824 podStartE2EDuration="23.954181887s" totalImagesPullingTime="19.447201063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:00.954071498 +0000 UTC m=+23.802390241" watchObservedRunningTime="2026-07-24 00:11:00.954181887 +0000 UTC m=+23.802500628"
Jul 24 00:11:00.977681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.977632    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hsbpp" podStartSLOduration=4.109883552 podStartE2EDuration="23.977613041s" totalImagesPullingTime="19.867729489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:00.977418499 +0000 UTC m=+23.825737240" watchObservedRunningTime="2026-07-24 00:11:00.977613041 +0000 UTC m=+23.825931784"
Jul 24 00:11:00.995967 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:00.995914    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-9.ec2.internal" podStartSLOduration=22.995899847 podStartE2EDuration="22.995899847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:00.995417751 +0000 UTC m=+23.843736495" watchObservedRunningTime="2026-07-24 00:11:00.995899847 +0000 UTC m=+23.844218590"
Jul 24 00:11:01.014046 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:01.013989    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7rcjj" podStartSLOduration=4.240087119 podStartE2EDuration="24.013975667s" totalImagesPullingTime="19.773888548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:01.013343311 +0000 UTC m=+23.861662051" watchObservedRunningTime="2026-07-24 00:11:01.013975667 +0000 UTC m=+23.862294408"
Jul 24 00:11:01.822397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:01.822286    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 00:11:01.867844 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:01.867807    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2" event={"ID":"10e19fc9-c926-4ecf-8748-2eff7762a0ff","Type":"ContainerStarted","Data":"2459b3c925e18734475d9d828a0a880773c97408365466eccc427a2f14aff00f"}
Jul 24 00:11:01.869786 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:01.869757    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal" event={"ID":"68d9b39a00920ba6f2d51237ba0e930b","Type":"ContainerStarted","Data":"c3fb7c8375d092124a001b249ba5a986b4f92bc834658ed44c606a51b83b7cf9"}
Jul 24 00:11:01.872046 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:01.872023    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8bw7w" event={"ID":"e12a6b24-c866-4f4d-b583-719700a08c6e","Type":"ContainerStarted","Data":"78dbaf5c2be026153d169f0f8ab68f34e506e548dc23c8f8f0ac1da6a686e94c"}
Jul 24 00:11:01.888150 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:01.888108    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-9.ec2.internal" podStartSLOduration=23.888096276 podStartE2EDuration="23.888096276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:01.88793039 +0000 UTC m=+24.736249130" watchObservedRunningTime="2026-07-24 00:11:01.888096276 +0000 UTC m=+24.736415017"
Jul 24 00:11:01.906197 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:01.906149    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8bw7w" podStartSLOduration=5.227990002 podStartE2EDuration="24.906135777s" totalImagesPullingTime="19.678145775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:01.905702611 +0000 UTC m=+24.754021354" watchObservedRunningTime="2026-07-24 00:11:01.906135777 +0000 UTC m=+24.754454517"
Jul 24 00:11:02.435514 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:02.435480    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:11:02.436109 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:02.436086    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:11:02.705171 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:02.705028    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T00:11:01.822330555Z","UUID":"0691a3ed-3db1-4d88-ac94-7e71d1cb3a28","Handler":null,"Name":"","Endpoint":""}
Jul 24 00:11:02.709551 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:02.709525    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 00:11:02.709701 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:02.709572    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 00:11:02.747172 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:02.746703    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:02.747172 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:02.746742    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:02.747172 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:02.746703    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:02.747172 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:02.746840    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:11:02.747172 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:02.746900    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:11:02.747172 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:02.746977    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:11:03.878830 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:03.878779    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2" event={"ID":"10e19fc9-c926-4ecf-8748-2eff7762a0ff","Type":"ContainerStarted","Data":"04f3fd6e6d4dd0c136427c3d95c8cc03e2b32f585fa8eb366fef845290a09c9a"}
Jul 24 00:11:03.882786 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:03.882748    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 00:11:03.882786 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:03.882765    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd" event={"ID":"7151f197-94ad-49a6-966a-1016bf6e4e9a","Type":"ContainerStarted","Data":"73330472675e2767d7a4bc4e46a1ffafcb2b510809ebba29062ef2bf75cb0a2c"}
Jul 24 00:11:03.887211 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:03.887189    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:11:03.887761 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:03.887734    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p7v47"
Jul 24 00:11:03.915547 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:03.915487    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfqj2" podStartSLOduration=4.5172077779999995 podStartE2EDuration="26.915467509s" totalImagesPullingTime="22.398259731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:03.898694282 +0000 UTC m=+26.747013024" watchObservedRunningTime="2026-07-24 00:11:03.915467509 +0000 UTC m=+26.763786251"
Jul 24 00:11:04.746932 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:04.746898    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:04.746932 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:04.746923    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:04.747174 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:04.746914    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:04.747174 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:04.747039    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:11:04.747174 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:04.747108    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:11:04.747304 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:04.747214    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:11:05.895039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:05.894847    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd" event={"ID":"7151f197-94ad-49a6-966a-1016bf6e4e9a","Type":"ContainerStarted","Data":"4c4450eadfba325cd0b66ddccbb5751581dfeb67f87cf89c876da9a2786673dd"}
Jul 24 00:11:05.926546 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:05.926501    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd" podStartSLOduration=9.139082647 podStartE2EDuration="28.926486787s" totalImagesPullingTime="19.78740414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:05.925971362 +0000 UTC m=+28.774290104" watchObservedRunningTime="2026-07-24 00:11:05.926486787 +0000 UTC m=+28.774805527"
Jul 24 00:11:06.571446 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:06.571417    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:11:06.746499 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:06.746462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:06.746705 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:06.746462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:06.746705 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:06.746600    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:11:06.746705 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:06.746462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:06.746861 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:06.746634    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:11:06.746861 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:06.746714    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:11:06.898842 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:06.898767    2574 generic.go:356] "Generic (PLEG): container finished" podID="acd4b7a0-11ee-40fb-ba36-9fb664430562" containerID="59e35148e1600bf2b6612dd15060a6a2960a954a140a09061605ad344af33107" exitCode=0
Jul 24 00:11:06.899430 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:06.898858    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vh7wc" event={"ID":"acd4b7a0-11ee-40fb-ba36-9fb664430562","Type":"ContainerDied","Data":"59e35148e1600bf2b6612dd15060a6a2960a954a140a09061605ad344af33107"}
Jul 24 00:11:06.899430 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:06.899342    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:11:06.899430 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:06.899419    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:11:06.914513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:06.914491    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:11:06.914747 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:06.914666    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:11:07.860846 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:07.860671    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-95fmv"]
Jul 24 00:11:07.860973 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:07.860925    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:07.861033 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:07.861010    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:11:07.864525 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:07.864400    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fmf4j"]
Jul 24 00:11:07.864525 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:07.864522    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:07.864703 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:07.864634    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:11:07.865058 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:07.865034    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j26n7"]
Jul 24 00:11:07.865149 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:07.865127    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:07.865258 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:07.865234    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:11:07.903139 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:07.903048    2574 generic.go:356] "Generic (PLEG): container finished" podID="acd4b7a0-11ee-40fb-ba36-9fb664430562" containerID="6be0c9e37aa8af29297244c1adfebe6d07445e7449b16f1477f33ea938339634" exitCode=0
Jul 24 00:11:07.903648 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:07.903147    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vh7wc" event={"ID":"acd4b7a0-11ee-40fb-ba36-9fb664430562","Type":"ContainerDied","Data":"6be0c9e37aa8af29297244c1adfebe6d07445e7449b16f1477f33ea938339634"}
Jul 24 00:11:09.747285 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:09.747254    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:09.747739 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:09.747254    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:09.747739 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:09.747392    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:11:09.747739 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:09.747408    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:09.747739 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:09.747530    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:11:09.747739 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:09.747621    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:11:09.910374 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:09.910346    2574 generic.go:356] "Generic (PLEG): container finished" podID="acd4b7a0-11ee-40fb-ba36-9fb664430562" containerID="d8855c8e4b503430653398514d9ac9dff52cdc5e56c1bbd002d62a9bd56ba6b5" exitCode=0
Jul 24 00:11:09.910533 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:09.910389    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vh7wc" event={"ID":"acd4b7a0-11ee-40fb-ba36-9fb664430562","Type":"ContainerDied","Data":"d8855c8e4b503430653398514d9ac9dff52cdc5e56c1bbd002d62a9bd56ba6b5"}
Jul 24 00:11:10.493423 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:10.493385    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:10.493626 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:10.493436    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:10.493626 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:10.493557    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:11:10.493626 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:10.493612    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret podName:a256a9ba-91cf-47ba-8e3b-a987081eac63 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:42.493598099 +0000 UTC m=+65.341916819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret") pod "global-pull-secret-syncer-j26n7" (UID: "a256a9ba-91cf-47ba-8e3b-a987081eac63") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:11:10.493626 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:10.493556    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:11:10.493902 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:10.493654    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs podName:1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:42.4936454 +0000 UTC m=+65.341964120 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs") pod "network-metrics-daemon-fmf4j" (UID: "1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:11:10.594483 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:10.594434    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2ts\" (UniqueName: \"kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts\") pod \"network-check-target-95fmv\" (UID: \"b3e62089-0435-442b-b800-fb8a90646c90\") " pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:10.594660 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:10.594632    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:11:10.594660 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:10.594658    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:11:10.594776 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:10.594674    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zw2ts for pod openshift-network-diagnostics/network-check-target-95fmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:11:10.594776 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:10.594737    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts podName:b3e62089-0435-442b-b800-fb8a90646c90 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:42.5947203 +0000 UTC m=+65.443039019 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zw2ts" (UniqueName: "kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts") pod "network-check-target-95fmv" (UID: "b3e62089-0435-442b-b800-fb8a90646c90") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:11:11.746880 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:11.746792    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:11.747350 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:11.746896    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fmv" podUID="b3e62089-0435-442b-b800-fb8a90646c90"
Jul 24 00:11:11.747350 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:11.746909    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:11.747350 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:11.746994    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j26n7" podUID="a256a9ba-91cf-47ba-8e3b-a987081eac63"
Jul 24 00:11:11.747350 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:11.747041    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:11.747350 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:11.747120    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmf4j" podUID="1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7"
Jul 24 00:11:12.965234 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:12.965004    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-9.ec2.internal" event="NodeReady"
Jul 24 00:11:12.965714 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:12.965386    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 00:11:13.040863 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.040832    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-546978b9f4-lqbqt"]
Jul 24 00:11:13.042817 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.042795    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.049160 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.049135    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2gxlr\""
Jul 24 00:11:13.049572 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.049546    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 00:11:13.050890 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.050732    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 00:11:13.056435 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.056414    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 00:11:13.064523 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.064498    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 00:11:13.073197 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.073171    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-546978b9f4-lqbqt"]
Jul 24 00:11:13.215552 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.215466    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.215552 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.215511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqst7\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-kube-api-access-tqst7\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.215762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.215570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-trusted-ca\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.215762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.215621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-installation-pull-secrets\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.215762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.215662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-image-registry-private-configuration\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.215762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.215691    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-registry-certificates\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.215762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.215728    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-bound-sa-token\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.216005 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.215826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77373f87-2556-48c0-b398-50315b64a2be-ca-trust-extracted\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.219568 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.219542    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jskz7"]
Jul 24 00:11:13.221423 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.221407    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vqjhl"]
Jul 24 00:11:13.221533 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.221517    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jskz7"
Jul 24 00:11:13.223202 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.223187    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.228968 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.227594    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 00:11:13.228968 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.227769    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dcszn\""
Jul 24 00:11:13.228968 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.227883    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 00:11:13.228968 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.228281    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qlhxg\""
Jul 24 00:11:13.228968 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.228560    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 00:11:13.228968 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.228626    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 00:11:13.228968 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.228801    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 00:11:13.265531 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.265417    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jskz7"]
Jul 24 00:11:13.266079 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.266053    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vqjhl"]
Jul 24 00:11:13.295528 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.295502    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-72lwd"]
Jul 24 00:11:13.297482 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.297460    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.301642 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.301618    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 00:11:13.302184 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.302161    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 00:11:13.302184 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.302179    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 00:11:13.302356 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.302200    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 00:11:13.302356 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.302169    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nsh42\""
Jul 24 00:11:13.316362 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316338    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909f0b40-b312-413e-864f-5acc0121e6d6-tmp-dir\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.316478 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316374    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqghx\" (UniqueName: \"kubernetes.io/projected/909f0b40-b312-413e-864f-5acc0121e6d6-kube-api-access-pqghx\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.316478 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316399    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.316478 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77373f87-2556-48c0-b398-50315b64a2be-ca-trust-extracted\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.316592 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.316592 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqst7\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-kube-api-access-tqst7\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.316592 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-trusted-ca\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.316704 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-installation-pull-secrets\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.316704 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-image-registry-private-configuration\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.316704 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.316685    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 00:11:13.316704 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/909f0b40-b312-413e-864f-5acc0121e6d6-config-volume\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.316704 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.316703    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546978b9f4-lqbqt: secret "image-registry-tls" not found
Jul 24 00:11:13.316933 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316718    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-registry-certificates\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.316933 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316738    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c04cdbd1-5fa0-4813-95eb-fc69a980dd2e-cert\") pod \"ingress-canary-jskz7\" (UID: \"c04cdbd1-5fa0-4813-95eb-fc69a980dd2e\") " pod="openshift-ingress-canary/ingress-canary-jskz7"
Jul 24 00:11:13.316933 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.316758    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls podName:77373f87-2556-48c0-b398-50315b64a2be nodeName:}" failed. No retries permitted until 2026-07-24 00:11:13.816737138 +0000 UTC m=+36.665055868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls") pod "image-registry-546978b9f4-lqbqt" (UID: "77373f87-2556-48c0-b398-50315b64a2be") : secret "image-registry-tls" not found
Jul 24 00:11:13.316933 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-bound-sa-token\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.317131 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.316921    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzqf\" (UniqueName: \"kubernetes.io/projected/c04cdbd1-5fa0-4813-95eb-fc69a980dd2e-kube-api-access-6pzqf\") pod \"ingress-canary-jskz7\" (UID: \"c04cdbd1-5fa0-4813-95eb-fc69a980dd2e\") " pod="openshift-ingress-canary/ingress-canary-jskz7"
Jul 24 00:11:13.317188 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.317130    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77373f87-2556-48c0-b398-50315b64a2be-ca-trust-extracted\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.317706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.317684    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-registry-certificates\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.317706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.317696    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-trusted-ca\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.322058 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.322030    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-installation-pull-secrets\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.322427 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.322401    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-image-registry-private-configuration\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.335397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.335372    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-72lwd"]
Jul 24 00:11:13.342305 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.342273    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ksmwx_6364a346-b501-4735-a5c3-97ee5b0120af/dns-node-resolver/0.log"
Jul 24 00:11:13.342634 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.342608    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqst7\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-kube-api-access-tqst7\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.343614 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.343588    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-bound-sa-token\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.417995 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.417956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/909f0b40-b312-413e-864f-5acc0121e6d6-config-volume\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.418196 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418009    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c04cdbd1-5fa0-4813-95eb-fc69a980dd2e-cert\") pod \"ingress-canary-jskz7\" (UID: \"c04cdbd1-5fa0-4813-95eb-fc69a980dd2e\") " pod="openshift-ingress-canary/ingress-canary-jskz7"
Jul 24 00:11:13.418196 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzqf\" (UniqueName: \"kubernetes.io/projected/c04cdbd1-5fa0-4813-95eb-fc69a980dd2e-kube-api-access-6pzqf\") pod \"ingress-canary-jskz7\" (UID: \"c04cdbd1-5fa0-4813-95eb-fc69a980dd2e\") " pod="openshift-ingress-canary/ingress-canary-jskz7"
Jul 24 00:11:13.418365 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418226    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909f0b40-b312-413e-864f-5acc0121e6d6-tmp-dir\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.418365 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418260    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqghx\" (UniqueName: \"kubernetes.io/projected/909f0b40-b312-413e-864f-5acc0121e6d6-kube-api-access-pqghx\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.418365 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418295    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/418b27c1-b425-4aed-a5e1-1f59b7a817b2-data-volume\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.418365 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.418548 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.418431    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 00:11:13.418548 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.418488    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls podName:909f0b40-b312-413e-864f-5acc0121e6d6 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:13.918470359 +0000 UTC m=+36.766789097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls") pod "dns-default-vqjhl" (UID: "909f0b40-b312-413e-864f-5acc0121e6d6") : secret "dns-default-metrics-tls" not found
Jul 24 00:11:13.418548 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418508    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.418548 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9tvv\" (UniqueName: \"kubernetes.io/projected/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-api-access-k9tvv\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.418745 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418565    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909f0b40-b312-413e-864f-5acc0121e6d6-tmp-dir\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.418745 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418626    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/909f0b40-b312-413e-864f-5acc0121e6d6-config-volume\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.418745 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418624    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.418745 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.418707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/418b27c1-b425-4aed-a5e1-1f59b7a817b2-crio-socket\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.420740 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.420717    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c04cdbd1-5fa0-4813-95eb-fc69a980dd2e-cert\") pod \"ingress-canary-jskz7\" (UID: \"c04cdbd1-5fa0-4813-95eb-fc69a980dd2e\") " pod="openshift-ingress-canary/ingress-canary-jskz7"
Jul 24 00:11:13.452980 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.452952    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqghx\" (UniqueName: \"kubernetes.io/projected/909f0b40-b312-413e-864f-5acc0121e6d6-kube-api-access-pqghx\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.453148 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.453131    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzqf\" (UniqueName: \"kubernetes.io/projected/c04cdbd1-5fa0-4813-95eb-fc69a980dd2e-kube-api-access-6pzqf\") pod \"ingress-canary-jskz7\" (UID: \"c04cdbd1-5fa0-4813-95eb-fc69a980dd2e\") " pod="openshift-ingress-canary/ingress-canary-jskz7"
Jul 24 00:11:13.504414 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.504335    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-5tm7n"]
Jul 24 00:11:13.507108 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.507088    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n"
Jul 24 00:11:13.509869 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.509845    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 24 00:11:13.509983 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.509898    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 24 00:11:13.510173 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.510139    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 24 00:11:13.510280 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.510174    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 24 00:11:13.510517 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.510498    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lkjc6\""
Jul 24 00:11:13.519512 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.519487    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/418b27c1-b425-4aed-a5e1-1f59b7a817b2-data-volume\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.519605 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.519526    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.519605 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.519543    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9tvv\" (UniqueName: \"kubernetes.io/projected/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-api-access-k9tvv\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.519605 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.519582    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.519772 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.519628    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/418b27c1-b425-4aed-a5e1-1f59b7a817b2-crio-socket\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.519824 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.519793    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 00:11:13.519877 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.519849    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls podName:418b27c1-b425-4aed-a5e1-1f59b7a817b2 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:14.019833283 +0000 UTC m=+36.868152019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72lwd" (UID: "418b27c1-b425-4aed-a5e1-1f59b7a817b2") : secret "insights-runtime-extractor-tls" not found
Jul 24 00:11:13.519944 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.519909    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 00:11:13.519996 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.519950    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/418b27c1-b425-4aed-a5e1-1f59b7a817b2-crio-socket\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.520073 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.520054    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/418b27c1-b425-4aed-a5e1-1f59b7a817b2-data-volume\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.520127 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.520100    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm podName:418b27c1-b425-4aed-a5e1-1f59b7a817b2 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:14.020080986 +0000 UTC m=+36.868399727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72lwd" (UID: "418b27c1-b425-4aed-a5e1-1f59b7a817b2") : configmap "kube-rbac-proxy" not found
Jul 24 00:11:13.520549 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.520531    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-5tm7n"]
Jul 24 00:11:13.534562 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.534538    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jskz7"
Jul 24 00:11:13.535650 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.535626    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9tvv\" (UniqueName: \"kubernetes.io/projected/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-api-access-k9tvv\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:13.620335 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.620280    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qgtj\" (UniqueName: \"kubernetes.io/projected/aa4ead67-d20f-4cae-a6fb-124d7fa5f947-kube-api-access-7qgtj\") pod \"service-ca-658d7b7c78-5tm7n\" (UID: \"aa4ead67-d20f-4cae-a6fb-124d7fa5f947\") " pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n"
Jul 24 00:11:13.620550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.620389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aa4ead67-d20f-4cae-a6fb-124d7fa5f947-signing-key\") pod \"service-ca-658d7b7c78-5tm7n\" (UID: \"aa4ead67-d20f-4cae-a6fb-124d7fa5f947\") " pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n"
Jul 24 00:11:13.620550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.620438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aa4ead67-d20f-4cae-a6fb-124d7fa5f947-signing-cabundle\") pod \"service-ca-658d7b7c78-5tm7n\" (UID: \"aa4ead67-d20f-4cae-a6fb-124d7fa5f947\") " pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n"
Jul 24 00:11:13.721770 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.721728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aa4ead67-d20f-4cae-a6fb-124d7fa5f947-signing-key\") pod \"service-ca-658d7b7c78-5tm7n\" (UID: \"aa4ead67-d20f-4cae-a6fb-124d7fa5f947\") " pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n"
Jul 24 00:11:13.721953 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.721830    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aa4ead67-d20f-4cae-a6fb-124d7fa5f947-signing-cabundle\") pod \"service-ca-658d7b7c78-5tm7n\" (UID: \"aa4ead67-d20f-4cae-a6fb-124d7fa5f947\") " pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n"
Jul 24 00:11:13.721953 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.721929    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qgtj\" (UniqueName: \"kubernetes.io/projected/aa4ead67-d20f-4cae-a6fb-124d7fa5f947-kube-api-access-7qgtj\") pod \"service-ca-658d7b7c78-5tm7n\" (UID: \"aa4ead67-d20f-4cae-a6fb-124d7fa5f947\") " pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n"
Jul 24 00:11:13.722636 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.722611    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aa4ead67-d20f-4cae-a6fb-124d7fa5f947-signing-cabundle\") pod \"service-ca-658d7b7c78-5tm7n\" (UID: \"aa4ead67-d20f-4cae-a6fb-124d7fa5f947\") " pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n"
Jul 24 00:11:13.724344 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.724304    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aa4ead67-d20f-4cae-a6fb-124d7fa5f947-signing-key\") pod \"service-ca-658d7b7c78-5tm7n\" (UID: \"aa4ead67-d20f-4cae-a6fb-124d7fa5f947\") " pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n"
Jul 24 00:11:13.729984 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.729963    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qgtj\" (UniqueName: \"kubernetes.io/projected/aa4ead67-d20f-4cae-a6fb-124d7fa5f947-kube-api-access-7qgtj\") pod \"service-ca-658d7b7c78-5tm7n\" (UID: \"aa4ead67-d20f-4cae-a6fb-124d7fa5f947\") " pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n"
Jul 24 00:11:13.747142 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.747105    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:13.747263 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.747212    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:13.747344 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.747300    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:13.750089 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.750067    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-79cjj\""
Jul 24 00:11:13.750363 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.750339    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 00:11:13.750446 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.750374    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rvqg9\""
Jul 24 00:11:13.750498 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.750452    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 00:11:13.750626 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.750611    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 00:11:13.750714 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.750640    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 00:11:13.817120 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.817084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n"
Jul 24 00:11:13.822489 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.822466    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:13.822604 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.822587    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 00:11:13.822652 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.822606    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546978b9f4-lqbqt: secret "image-registry-tls" not found
Jul 24 00:11:13.822697 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.822676    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls podName:77373f87-2556-48c0-b398-50315b64a2be nodeName:}" failed. No retries permitted until 2026-07-24 00:11:14.82265774 +0000 UTC m=+37.670976479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls") pod "image-registry-546978b9f4-lqbqt" (UID: "77373f87-2556-48c0-b398-50315b64a2be") : secret "image-registry-tls" not found
Jul 24 00:11:13.923326 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:13.923269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:13.923501 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.923430    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 00:11:13.923562 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:13.923509    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls podName:909f0b40-b312-413e-864f-5acc0121e6d6 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:14.923488558 +0000 UTC m=+37.771807303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls") pod "dns-default-vqjhl" (UID: "909f0b40-b312-413e-864f-5acc0121e6d6") : secret "dns-default-metrics-tls" not found
Jul 24 00:11:14.024127 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:14.024043    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:14.024127 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:14.024091    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:14.024609 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:14.024196    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 00:11:14.024609 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:14.024213    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 00:11:14.024609 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:14.024263    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm podName:418b27c1-b425-4aed-a5e1-1f59b7a817b2 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:15.02424932 +0000 UTC m=+37.872568057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72lwd" (UID: "418b27c1-b425-4aed-a5e1-1f59b7a817b2") : configmap "kube-rbac-proxy" not found
Jul 24 00:11:14.024609 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:14.024277    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls podName:418b27c1-b425-4aed-a5e1-1f59b7a817b2 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:15.024271723 +0000 UTC m=+37.872590442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72lwd" (UID: "418b27c1-b425-4aed-a5e1-1f59b7a817b2") : secret "insights-runtime-extractor-tls" not found
Jul 24 00:11:14.046071 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:14.046048    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kdddd_f5b7f667-f9ea-4deb-9cb8-362fe65e017e/node-ca/0.log"
Jul 24 00:11:14.461632 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:14.460450    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-5tm7n"]
Jul 24 00:11:14.463608 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:14.463582    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4ead67_d20f_4cae_a6fb_124d7fa5f947.slice/crio-5a4aa195b0a6bda4fe76d69f2cb1c5be095b95bcaf3d0e6b19992dc905f2a097 WatchSource:0}: Error finding container 5a4aa195b0a6bda4fe76d69f2cb1c5be095b95bcaf3d0e6b19992dc905f2a097: Status 404 returned error can't find the container with id 5a4aa195b0a6bda4fe76d69f2cb1c5be095b95bcaf3d0e6b19992dc905f2a097
Jul 24 00:11:14.470156 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:14.470128    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jskz7"]
Jul 24 00:11:14.474050 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:14.474021    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04cdbd1_5fa0_4813_95eb_fc69a980dd2e.slice/crio-d06b370fea2c71fdb87a2e1e14d85eb7a1dfaf564daf9f4cab6370898a850ff8 WatchSource:0}: Error finding container d06b370fea2c71fdb87a2e1e14d85eb7a1dfaf564daf9f4cab6370898a850ff8: Status 404 returned error can't find the container with id d06b370fea2c71fdb87a2e1e14d85eb7a1dfaf564daf9f4cab6370898a850ff8
Jul 24 00:11:14.830872 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:14.830841    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:14.831044 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:14.830972    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 00:11:14.831044 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:14.830984    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546978b9f4-lqbqt: secret "image-registry-tls" not found
Jul 24 00:11:14.831044 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:14.831033    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls podName:77373f87-2556-48c0-b398-50315b64a2be nodeName:}" failed. No retries permitted until 2026-07-24 00:11:16.831018748 +0000 UTC m=+39.679337468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls") pod "image-registry-546978b9f4-lqbqt" (UID: "77373f87-2556-48c0-b398-50315b64a2be") : secret "image-registry-tls" not found
Jul 24 00:11:14.923567 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:14.923525    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jskz7" event={"ID":"c04cdbd1-5fa0-4813-95eb-fc69a980dd2e","Type":"ContainerStarted","Data":"d06b370fea2c71fdb87a2e1e14d85eb7a1dfaf564daf9f4cab6370898a850ff8"}
Jul 24 00:11:14.924586 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:14.924562    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n" event={"ID":"aa4ead67-d20f-4cae-a6fb-124d7fa5f947","Type":"ContainerStarted","Data":"5a4aa195b0a6bda4fe76d69f2cb1c5be095b95bcaf3d0e6b19992dc905f2a097"}
Jul 24 00:11:14.926908 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:14.926886    2574 generic.go:356] "Generic (PLEG): container finished" podID="acd4b7a0-11ee-40fb-ba36-9fb664430562" containerID="08f5455a6d12eb6757d71132de572d24c7691d248cb4cd28ad392a461166bbd2" exitCode=0
Jul 24 00:11:14.927010 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:14.926938    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vh7wc" event={"ID":"acd4b7a0-11ee-40fb-ba36-9fb664430562","Type":"ContainerDied","Data":"08f5455a6d12eb6757d71132de572d24c7691d248cb4cd28ad392a461166bbd2"}
Jul 24 00:11:14.931384 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:14.931366    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:14.931510 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:14.931492    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 00:11:14.931556 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:14.931547    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls podName:909f0b40-b312-413e-864f-5acc0121e6d6 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:16.93153183 +0000 UTC m=+39.779850549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls") pod "dns-default-vqjhl" (UID: "909f0b40-b312-413e-864f-5acc0121e6d6") : secret "dns-default-metrics-tls" not found
Jul 24 00:11:15.033263 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:15.033080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:15.033263 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:15.033154    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 00:11:15.033263 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:15.033177    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:15.033263 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:15.033207    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls podName:418b27c1-b425-4aed-a5e1-1f59b7a817b2 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:17.033188661 +0000 UTC m=+39.881507406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72lwd" (UID: "418b27c1-b425-4aed-a5e1-1f59b7a817b2") : secret "insights-runtime-extractor-tls" not found
Jul 24 00:11:15.033263 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:15.033233    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 00:11:15.033263 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:15.033268    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm podName:418b27c1-b425-4aed-a5e1-1f59b7a817b2 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:17.033259218 +0000 UTC m=+39.881577937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72lwd" (UID: "418b27c1-b425-4aed-a5e1-1f59b7a817b2") : configmap "kube-rbac-proxy" not found
Jul 24 00:11:15.933199 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:15.933163    2574 generic.go:356] "Generic (PLEG): container finished" podID="acd4b7a0-11ee-40fb-ba36-9fb664430562" containerID="14b1029f1e7e96d0b0a8d1e35a7f67b234dbfe2dd5705c1b5ab97b9ce1f03895" exitCode=0
Jul 24 00:11:15.933403 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:15.933225    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vh7wc" event={"ID":"acd4b7a0-11ee-40fb-ba36-9fb664430562","Type":"ContainerDied","Data":"14b1029f1e7e96d0b0a8d1e35a7f67b234dbfe2dd5705c1b5ab97b9ce1f03895"}
Jul 24 00:11:16.847397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:16.847356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:16.847910 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:16.847542    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 00:11:16.847910 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:16.847565    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546978b9f4-lqbqt: secret "image-registry-tls" not found
Jul 24 00:11:16.847910 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:16.847638    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls podName:77373f87-2556-48c0-b398-50315b64a2be nodeName:}" failed. No retries permitted until 2026-07-24 00:11:20.847615129 +0000 UTC m=+43.695933861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls") pod "image-registry-546978b9f4-lqbqt" (UID: "77373f87-2556-48c0-b398-50315b64a2be") : secret "image-registry-tls" not found
Jul 24 00:11:16.940302 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:16.940237    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vh7wc" event={"ID":"acd4b7a0-11ee-40fb-ba36-9fb664430562","Type":"ContainerStarted","Data":"ef27acbe226db467cf1b27b39bca4f7bf5cef93c26ebcf92bf2c094d5a6a6175"}
Jul 24 00:11:16.948375 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:16.948335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:16.948524 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:16.948419    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 00:11:16.948524 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:16.948472    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls podName:909f0b40-b312-413e-864f-5acc0121e6d6 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:20.948455402 +0000 UTC m=+43.796774128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls") pod "dns-default-vqjhl" (UID: "909f0b40-b312-413e-864f-5acc0121e6d6") : secret "dns-default-metrics-tls" not found
Jul 24 00:11:16.967986 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:16.967935    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vh7wc" podStartSLOduration=9.29692243 podStartE2EDuration="39.967918382s" totalImagesPullingTime="30.670995952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:16.965433151 +0000 UTC m=+39.813751892" watchObservedRunningTime="2026-07-24 00:11:16.967918382 +0000 UTC m=+39.816237120"
Jul 24 00:11:17.049738 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:17.049691    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:17.049920 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:17.049776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:17.049920 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:17.049851    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 00:11:17.050018 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:17.049961    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls podName:418b27c1-b425-4aed-a5e1-1f59b7a817b2 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:21.049934442 +0000 UTC m=+43.898253184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72lwd" (UID: "418b27c1-b425-4aed-a5e1-1f59b7a817b2") : secret "insights-runtime-extractor-tls" not found
Jul 24 00:11:17.050559 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:17.050529    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/418b27c1-b425-4aed-a5e1-1f59b7a817b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:17.943988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:17.943949    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n" event={"ID":"aa4ead67-d20f-4cae-a6fb-124d7fa5f947","Type":"ContainerStarted","Data":"f68fb9729fcc5b2f9df7964d9b8ed7e72510d9f516ed40d7772827084246f16a"}
Jul 24 00:11:17.945374 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:17.945347    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jskz7" event={"ID":"c04cdbd1-5fa0-4813-95eb-fc69a980dd2e","Type":"ContainerStarted","Data":"9292a82178469869a43cf4db4f131ceda406e806f59bebed9efd3b81374d46a1"}
Jul 24 00:11:17.964455 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:17.964402    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-5tm7n" podStartSLOduration=2.017545111 podStartE2EDuration="4.964386308s" totalImagesPullingTime="2.946841197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:17.963619806 +0000 UTC m=+40.811938552" watchObservedRunningTime="2026-07-24 00:11:17.964386308 +0000 UTC m=+40.812705049"
Jul 24 00:11:17.980462 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:17.980414    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jskz7" podStartSLOduration=2.036757568 podStartE2EDuration="4.980396912s" totalImagesPullingTime="2.943639344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:17.980024141 +0000 UTC m=+40.828342881" watchObservedRunningTime="2026-07-24 00:11:17.980396912 +0000 UTC m=+40.828715653"
Jul 24 00:11:20.878402 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:20.878365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:20.878791 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:20.878525    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 00:11:20.878791 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:20.878545    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546978b9f4-lqbqt: secret "image-registry-tls" not found
Jul 24 00:11:20.878791 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:20.878597    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls podName:77373f87-2556-48c0-b398-50315b64a2be nodeName:}" failed. No retries permitted until 2026-07-24 00:11:28.878582076 +0000 UTC m=+51.726900794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls") pod "image-registry-546978b9f4-lqbqt" (UID: "77373f87-2556-48c0-b398-50315b64a2be") : secret "image-registry-tls" not found
Jul 24 00:11:20.979033 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:20.978995    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:20.979165 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:20.979144    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 00:11:20.979214 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:20.979204    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls podName:909f0b40-b312-413e-864f-5acc0121e6d6 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:28.979188751 +0000 UTC m=+51.827507469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls") pod "dns-default-vqjhl" (UID: "909f0b40-b312-413e-864f-5acc0121e6d6") : secret "dns-default-metrics-tls" not found
Jul 24 00:11:21.079429 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:21.079397    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:21.079995 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:21.079964    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 00:11:21.080196 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:11:21.080182    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls podName:418b27c1-b425-4aed-a5e1-1f59b7a817b2 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:29.080159397 +0000 UTC m=+51.928478119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72lwd" (UID: "418b27c1-b425-4aed-a5e1-1f59b7a817b2") : secret "insights-runtime-extractor-tls" not found
Jul 24 00:11:28.941955 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:28.941919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:28.945482 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:28.945458    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls\") pod \"image-registry-546978b9f4-lqbqt\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") " pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:28.954250 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:28.954232    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:29.055334 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.052416    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:29.068559 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.068474    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/909f0b40-b312-413e-864f-5acc0121e6d6-metrics-tls\") pod \"dns-default-vqjhl\" (UID: \"909f0b40-b312-413e-864f-5acc0121e6d6\") " pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:29.108748 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.108718    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-546978b9f4-lqbqt"]
Jul 24 00:11:29.112070 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:29.112036    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77373f87_2556_48c0_b398_50315b64a2be.slice/crio-f77764b7d97a197f44df5be1610312c402b584263536c4bb11a00c8317dcd6dc WatchSource:0}: Error finding container f77764b7d97a197f44df5be1610312c402b584263536c4bb11a00c8317dcd6dc: Status 404 returned error can't find the container with id f77764b7d97a197f44df5be1610312c402b584263536c4bb11a00c8317dcd6dc
Jul 24 00:11:29.140996 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.140976    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:29.153461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.153440    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:29.155478 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.155460    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/418b27c1-b425-4aed-a5e1-1f59b7a817b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72lwd\" (UID: \"418b27c1-b425-4aed-a5e1-1f59b7a817b2\") " pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:29.208472 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.208448    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-72lwd"
Jul 24 00:11:29.278682 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.278654    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vqjhl"]
Jul 24 00:11:29.281004 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:29.280976    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909f0b40_b312_413e_864f_5acc0121e6d6.slice/crio-9c15aa506c68a3e6a3be97265b7a1f631647dde7ae785ffd27bda36df1d54f67 WatchSource:0}: Error finding container 9c15aa506c68a3e6a3be97265b7a1f631647dde7ae785ffd27bda36df1d54f67: Status 404 returned error can't find the container with id 9c15aa506c68a3e6a3be97265b7a1f631647dde7ae785ffd27bda36df1d54f67
Jul 24 00:11:29.345813 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.345786    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-72lwd"]
Jul 24 00:11:29.348901 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:29.348872    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418b27c1_b425_4aed_a5e1_1f59b7a817b2.slice/crio-dcfb5eb7405d7e8a475563acd36f1405d5574b26014fb163fa89a3e8342ff7f4 WatchSource:0}: Error finding container dcfb5eb7405d7e8a475563acd36f1405d5574b26014fb163fa89a3e8342ff7f4: Status 404 returned error can't find the container with id dcfb5eb7405d7e8a475563acd36f1405d5574b26014fb163fa89a3e8342ff7f4
Jul 24 00:11:29.978647 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.978609    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72lwd" event={"ID":"418b27c1-b425-4aed-a5e1-1f59b7a817b2","Type":"ContainerStarted","Data":"dcfb5eb7405d7e8a475563acd36f1405d5574b26014fb163fa89a3e8342ff7f4"}
Jul 24 00:11:29.979873 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.979785    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqjhl" event={"ID":"909f0b40-b312-413e-864f-5acc0121e6d6","Type":"ContainerStarted","Data":"9c15aa506c68a3e6a3be97265b7a1f631647dde7ae785ffd27bda36df1d54f67"}
Jul 24 00:11:29.981837 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.981349    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt" event={"ID":"77373f87-2556-48c0-b398-50315b64a2be","Type":"ContainerStarted","Data":"d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87"}
Jul 24 00:11:29.981837 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.981395    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt" event={"ID":"77373f87-2556-48c0-b398-50315b64a2be","Type":"ContainerStarted","Data":"f77764b7d97a197f44df5be1610312c402b584263536c4bb11a00c8317dcd6dc"}
Jul 24 00:11:29.981837 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:29.981674    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:31.990855 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:31.989974    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqjhl" event={"ID":"909f0b40-b312-413e-864f-5acc0121e6d6","Type":"ContainerStarted","Data":"b8316957fbd9d5f2d7dff2b768b3cc2538c10a225c12dee8a2b595b1356767cb"}
Jul 24 00:11:31.990855 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:31.990054    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqjhl" event={"ID":"909f0b40-b312-413e-864f-5acc0121e6d6","Type":"ContainerStarted","Data":"77503653b8a2cf37882294659e88548545a3cd61df5ec0e0da78409a3b31c4b5"}
Jul 24 00:11:31.990855 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:31.990607    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:32.008457 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:32.008394    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt" podStartSLOduration=37.008375921 podStartE2EDuration="37.008375921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:30.009353512 +0000 UTC m=+52.857672260" watchObservedRunningTime="2026-07-24 00:11:32.008375921 +0000 UTC m=+54.856694643"
Jul 24 00:11:35.000250 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.000213    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72lwd" event={"ID":"418b27c1-b425-4aed-a5e1-1f59b7a817b2","Type":"ContainerStarted","Data":"bacab7b46b5f0a3f298e2f7bbcec7e7a2755617473f04b32fad7b9c20f5f3e4b"}
Jul 24 00:11:35.335520 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.335432    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vqjhl" podStartSLOduration=19.978787069 podStartE2EDuration="22.335399548s" totalImagesPullingTime="2.356612479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:32.007614984 +0000 UTC m=+54.855933736" watchObservedRunningTime="2026-07-24 00:11:35.335399548 +0000 UTC m=+58.183718291"
Jul 24 00:11:35.337388 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.337358    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm"]
Jul 24 00:11:35.339550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.339514    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"]
Jul 24 00:11:35.339643 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.339609    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm"
Jul 24 00:11:35.341903 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.341882    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"]
Jul 24 00:11:35.342083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.342058    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.342881 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.342860    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 24 00:11:35.342979 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.342865    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 00:11:35.343037 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.342986    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rbm6q\""
Jul 24 00:11:35.343111 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.343093    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 00:11:35.343242 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.343224    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 00:11:35.343973 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.343954    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:35.344508 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.344485    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 24 00:11:35.344952 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.344932    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 24 00:11:35.345887 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.345860    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 24 00:11:35.345988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.345951    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 24 00:11:35.348593 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.348571    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 24 00:11:35.359769 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.359723    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm"]
Jul 24 00:11:35.366423 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.361132    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"]
Jul 24 00:11:35.366423 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.363692    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"]
Jul 24 00:11:35.398905 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.398511    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-546978b9f4-lqbqt"]
Jul 24 00:11:35.408716 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.408683    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2j8b\" (UniqueName: \"kubernetes.io/projected/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-kube-api-access-m2j8b\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.408868 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.408733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpf74\" (UniqueName: \"kubernetes.io/projected/cf5cfa3d-5b43-441d-b437-7741c5e87560-kube-api-access-mpf74\") pod \"klusterlet-addon-workmgr-747d9d96b4-4dkzw\" (UID: \"cf5cfa3d-5b43-441d-b437-7741c5e87560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:35.408868 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.408774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-ca\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.408868 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.408798    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cf5cfa3d-5b43-441d-b437-7741c5e87560-klusterlet-config\") pod \"klusterlet-addon-workmgr-747d9d96b4-4dkzw\" (UID: \"cf5cfa3d-5b43-441d-b437-7741c5e87560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:35.408868 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.408847    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf5cfa3d-5b43-441d-b437-7741c5e87560-tmp\") pod \"klusterlet-addon-workmgr-747d9d96b4-4dkzw\" (UID: \"cf5cfa3d-5b43-441d-b437-7741c5e87560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:35.409137 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.408874    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-hub\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.409137 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.408909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.409137 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.408941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95829203-4e33-4406-8223-e860f545755b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm\" (UID: \"95829203-4e33-4406-8223-e860f545755b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm"
Jul 24 00:11:35.409137 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.408963    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.409137 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.409017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxv7\" (UniqueName: \"kubernetes.io/projected/95829203-4e33-4406-8223-e860f545755b-kube-api-access-5jxv7\") pod \"managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm\" (UID: \"95829203-4e33-4406-8223-e860f545755b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm"
Jul 24 00:11:35.409137 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.409111    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.455183 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.455146    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66ffc854c8-5t479"]
Jul 24 00:11:35.457103 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.457086    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.474504 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.474479    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66ffc854c8-5t479"]
Jul 24 00:11:35.510327 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510275    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cf5cfa3d-5b43-441d-b437-7741c5e87560-klusterlet-config\") pod \"klusterlet-addon-workmgr-747d9d96b4-4dkzw\" (UID: \"cf5cfa3d-5b43-441d-b437-7741c5e87560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:35.510510 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510361    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/134e7585-1ffa-4f79-bc7d-bc599bc98893-ca-trust-extracted\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.510510 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/134e7585-1ffa-4f79-bc7d-bc599bc98893-registry-tls\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.510510 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510437    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf5cfa3d-5b43-441d-b437-7741c5e87560-tmp\") pod \"klusterlet-addon-workmgr-747d9d96b4-4dkzw\" (UID: \"cf5cfa3d-5b43-441d-b437-7741c5e87560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:35.510510 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-hub\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.510510 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/134e7585-1ffa-4f79-bc7d-bc599bc98893-installation-pull-secrets\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.510826 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.510826 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510557    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/134e7585-1ffa-4f79-bc7d-bc599bc98893-bound-sa-token\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.510826 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/134e7585-1ffa-4f79-bc7d-bc599bc98893-trusted-ca\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.510826 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95829203-4e33-4406-8223-e860f545755b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm\" (UID: \"95829203-4e33-4406-8223-e860f545755b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm"
Jul 24 00:11:35.510826 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.511078 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510910    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/134e7585-1ffa-4f79-bc7d-bc599bc98893-registry-certificates\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.511078 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/134e7585-1ffa-4f79-bc7d-bc599bc98893-image-registry-private-configuration\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.511078 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.510984    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf5cfa3d-5b43-441d-b437-7741c5e87560-tmp\") pod \"klusterlet-addon-workmgr-747d9d96b4-4dkzw\" (UID: \"cf5cfa3d-5b43-441d-b437-7741c5e87560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:35.511078 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.511006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxv7\" (UniqueName: \"kubernetes.io/projected/95829203-4e33-4406-8223-e860f545755b-kube-api-access-5jxv7\") pod \"managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm\" (UID: \"95829203-4e33-4406-8223-e860f545755b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm"
Jul 24 00:11:35.511078 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.511071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.511717 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.511109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2j8b\" (UniqueName: \"kubernetes.io/projected/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-kube-api-access-m2j8b\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.511717 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.511144    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpf74\" (UniqueName: \"kubernetes.io/projected/cf5cfa3d-5b43-441d-b437-7741c5e87560-kube-api-access-mpf74\") pod \"klusterlet-addon-workmgr-747d9d96b4-4dkzw\" (UID: \"cf5cfa3d-5b43-441d-b437-7741c5e87560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:35.511717 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.511183    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p92h\" (UniqueName: \"kubernetes.io/projected/134e7585-1ffa-4f79-bc7d-bc599bc98893-kube-api-access-2p92h\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.511717 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.511218    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-ca\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.511717 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.511354    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.513836 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.513810    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95829203-4e33-4406-8223-e860f545755b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm\" (UID: \"95829203-4e33-4406-8223-e860f545755b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm"
Jul 24 00:11:35.514360 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.514338    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-hub\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.514501 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.514477    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.514721 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.514684    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-ca\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.514961 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.514930    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.515173 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.515156    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cf5cfa3d-5b43-441d-b437-7741c5e87560-klusterlet-config\") pod \"klusterlet-addon-workmgr-747d9d96b4-4dkzw\" (UID: \"cf5cfa3d-5b43-441d-b437-7741c5e87560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:35.521496 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.521451    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpf74\" (UniqueName: \"kubernetes.io/projected/cf5cfa3d-5b43-441d-b437-7741c5e87560-kube-api-access-mpf74\") pod \"klusterlet-addon-workmgr-747d9d96b4-4dkzw\" (UID: \"cf5cfa3d-5b43-441d-b437-7741c5e87560\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:35.521599 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.521564    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2j8b\" (UniqueName: \"kubernetes.io/projected/bbc389d0-fa5c-4620-93cf-335b25bd9e2a-kube-api-access-m2j8b\") pod \"cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd\" (UID: \"bbc389d0-fa5c-4620-93cf-335b25bd9e2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.521787 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.521770    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxv7\" (UniqueName: \"kubernetes.io/projected/95829203-4e33-4406-8223-e860f545755b-kube-api-access-5jxv7\") pod \"managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm\" (UID: \"95829203-4e33-4406-8223-e860f545755b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm"
Jul 24 00:11:35.612607 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.612509    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p92h\" (UniqueName: \"kubernetes.io/projected/134e7585-1ffa-4f79-bc7d-bc599bc98893-kube-api-access-2p92h\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.612607 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.612563    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/134e7585-1ffa-4f79-bc7d-bc599bc98893-ca-trust-extracted\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.612812 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.612623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/134e7585-1ffa-4f79-bc7d-bc599bc98893-registry-tls\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.612812 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.612691    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/134e7585-1ffa-4f79-bc7d-bc599bc98893-installation-pull-secrets\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.612812 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.612736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/134e7585-1ffa-4f79-bc7d-bc599bc98893-bound-sa-token\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.612812 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.612773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/134e7585-1ffa-4f79-bc7d-bc599bc98893-trusted-ca\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.613000 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.612825    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/134e7585-1ffa-4f79-bc7d-bc599bc98893-registry-certificates\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.613000 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.612856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/134e7585-1ffa-4f79-bc7d-bc599bc98893-image-registry-private-configuration\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.614257 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.613574    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/134e7585-1ffa-4f79-bc7d-bc599bc98893-ca-trust-extracted\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.614257 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.614201    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/134e7585-1ffa-4f79-bc7d-bc599bc98893-trusted-ca\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.614526 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.614257    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/134e7585-1ffa-4f79-bc7d-bc599bc98893-registry-certificates\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.615466 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.615447    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/134e7585-1ffa-4f79-bc7d-bc599bc98893-installation-pull-secrets\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.615612 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.615596    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/134e7585-1ffa-4f79-bc7d-bc599bc98893-registry-tls\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.615758 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.615732    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/134e7585-1ffa-4f79-bc7d-bc599bc98893-image-registry-private-configuration\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.628402 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.628373    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/134e7585-1ffa-4f79-bc7d-bc599bc98893-bound-sa-token\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.628582 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.628564    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p92h\" (UniqueName: \"kubernetes.io/projected/134e7585-1ffa-4f79-bc7d-bc599bc98893-kube-api-access-2p92h\") pod \"image-registry-66ffc854c8-5t479\" (UID: \"134e7585-1ffa-4f79-bc7d-bc599bc98893\") " pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.658593 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.658566    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm"
Jul 24 00:11:35.672472 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.672450    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"
Jul 24 00:11:35.681449 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.681428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:35.776422 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.774837    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:35.810500 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.810441    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm"]
Jul 24 00:11:35.816056 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:35.816009    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95829203_4e33_4406_8223_e860f545755b.slice/crio-1e96f31fc24a6cdf5c6ff1b8447555f6a33a89c709273d7c8994d243dc985d10 WatchSource:0}: Error finding container 1e96f31fc24a6cdf5c6ff1b8447555f6a33a89c709273d7c8994d243dc985d10: Status 404 returned error can't find the container with id 1e96f31fc24a6cdf5c6ff1b8447555f6a33a89c709273d7c8994d243dc985d10
Jul 24 00:11:35.839077 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.839044    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd"]
Jul 24 00:11:35.847519 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:35.847483    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbc389d0_fa5c_4620_93cf_335b25bd9e2a.slice/crio-bcf6db69b6651dba9061442ca4bc9c4bb3788994d334348347f11d3993268d7f WatchSource:0}: Error finding container bcf6db69b6651dba9061442ca4bc9c4bb3788994d334348347f11d3993268d7f: Status 404 returned error can't find the container with id bcf6db69b6651dba9061442ca4bc9c4bb3788994d334348347f11d3993268d7f
Jul 24 00:11:35.855406 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.855380    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"]
Jul 24 00:11:35.858506 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:35.858479    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5cfa3d_5b43_441d_b437_7741c5e87560.slice/crio-78f2f81bd7bb13c972ae02ca8b005c17fafb04c7a807ef070da54e86a01b14d9 WatchSource:0}: Error finding container 78f2f81bd7bb13c972ae02ca8b005c17fafb04c7a807ef070da54e86a01b14d9: Status 404 returned error can't find the container with id 78f2f81bd7bb13c972ae02ca8b005c17fafb04c7a807ef070da54e86a01b14d9
Jul 24 00:11:35.929999 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:35.929969    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66ffc854c8-5t479"]
Jul 24 00:11:35.933272 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:35.933215    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134e7585_1ffa_4f79_bc7d_bc599bc98893.slice/crio-821a58f53ffb369ef67d39514bfa9a283fe99e4d7313601e1920f097901822c5 WatchSource:0}: Error finding container 821a58f53ffb369ef67d39514bfa9a283fe99e4d7313601e1920f097901822c5: Status 404 returned error can't find the container with id 821a58f53ffb369ef67d39514bfa9a283fe99e4d7313601e1920f097901822c5
Jul 24 00:11:36.005156 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:36.005124    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw" event={"ID":"cf5cfa3d-5b43-441d-b437-7741c5e87560","Type":"ContainerStarted","Data":"78f2f81bd7bb13c972ae02ca8b005c17fafb04c7a807ef070da54e86a01b14d9"}
Jul 24 00:11:36.006665 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:36.006635    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd" event={"ID":"bbc389d0-fa5c-4620-93cf-335b25bd9e2a","Type":"ContainerStarted","Data":"bcf6db69b6651dba9061442ca4bc9c4bb3788994d334348347f11d3993268d7f"}
Jul 24 00:11:36.008546 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:36.008522    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66ffc854c8-5t479" event={"ID":"134e7585-1ffa-4f79-bc7d-bc599bc98893","Type":"ContainerStarted","Data":"821a58f53ffb369ef67d39514bfa9a283fe99e4d7313601e1920f097901822c5"}
Jul 24 00:11:36.009909 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:36.009875    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm" event={"ID":"95829203-4e33-4406-8223-e860f545755b","Type":"ContainerStarted","Data":"1e96f31fc24a6cdf5c6ff1b8447555f6a33a89c709273d7c8994d243dc985d10"}
Jul 24 00:11:37.015070 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:37.015029    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66ffc854c8-5t479" event={"ID":"134e7585-1ffa-4f79-bc7d-bc599bc98893","Type":"ContainerStarted","Data":"aafd10979ae0b46b854ab17eed67781c73839eb1c0a6a7584a50f8b240fe4fd3"}
Jul 24 00:11:37.015522 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:37.015143    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:37.036163 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:37.036116    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66ffc854c8-5t479" podStartSLOduration=2.036098953 podStartE2EDuration="2.036098953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:37.03507499 +0000 UTC m=+59.883393746" watchObservedRunningTime="2026-07-24 00:11:37.036098953 +0000 UTC m=+59.884417694"
Jul 24 00:11:38.924789 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:38.923628    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlsrd"
Jul 24 00:11:40.036712 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:40.036678    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72lwd" event={"ID":"418b27c1-b425-4aed-a5e1-1f59b7a817b2","Type":"ContainerStarted","Data":"435d82f6a855df9d2b5f4f32dc36e784e99754b8b23edc4375e3f3ac6359e3fe"}
Jul 24 00:11:42.584126 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.584085    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:42.584581 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.584193    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:42.586918 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.586896    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 00:11:42.587055 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.586970    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 00:11:42.597851 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.597825    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7-metrics-certs\") pod \"network-metrics-daemon-fmf4j\" (UID: \"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7\") " pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:42.597946 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.597864    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a256a9ba-91cf-47ba-8e3b-a987081eac63-original-pull-secret\") pod \"global-pull-secret-syncer-j26n7\" (UID: \"a256a9ba-91cf-47ba-8e3b-a987081eac63\") " pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:42.684663 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.684626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2ts\" (UniqueName: \"kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts\") pod \"network-check-target-95fmv\" (UID: \"b3e62089-0435-442b-b800-fb8a90646c90\") " pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:42.687696 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.687676    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 00:11:42.697628 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.697593    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 00:11:42.708283 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.708263    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw2ts\" (UniqueName: \"kubernetes.io/projected/b3e62089-0435-442b-b800-fb8a90646c90-kube-api-access-zw2ts\") pod \"network-check-target-95fmv\" (UID: \"b3e62089-0435-442b-b800-fb8a90646c90\") " pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:42.861149 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.861071    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-79cjj\""
Jul 24 00:11:42.865346 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.865303    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j26n7"
Jul 24 00:11:42.869105 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.869083    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:42.874082 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.874063    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rvqg9\""
Jul 24 00:11:42.882383 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.882363    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmf4j"
Jul 24 00:11:42.997460 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:42.997431    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vqjhl"
Jul 24 00:11:45.405443 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:45.405410    2574 patch_prober.go:28] interesting pod/image-registry-546978b9f4-lqbqt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 24 00:11:45.405798 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:45.405479    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt" podUID="77373f87-2556-48c0-b398-50315b64a2be" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 00:11:45.525774 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:45.524945    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fmf4j"]
Jul 24 00:11:45.529620 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:45.529575    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caa4b14_4b7d_43c5_b0c6_42a1b70e02f7.slice/crio-173a72c32c1547fbfa918c9a36332be1b2f1eca52d8a9fc2ec59c6093ac85008 WatchSource:0}: Error finding container 173a72c32c1547fbfa918c9a36332be1b2f1eca52d8a9fc2ec59c6093ac85008: Status 404 returned error can't find the container with id 173a72c32c1547fbfa918c9a36332be1b2f1eca52d8a9fc2ec59c6093ac85008
Jul 24 00:11:45.547994 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:45.547956    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j26n7"]
Jul 24 00:11:45.565791 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:45.565763    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-95fmv"]
Jul 24 00:11:45.723346 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:45.723302    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda256a9ba_91cf_47ba_8e3b_a987081eac63.slice/crio-a3b8cb1864480b0e208cc3d4bbc319bbce3c39e53b81dcf95430f780eb57911c WatchSource:0}: Error finding container a3b8cb1864480b0e208cc3d4bbc319bbce3c39e53b81dcf95430f780eb57911c: Status 404 returned error can't find the container with id a3b8cb1864480b0e208cc3d4bbc319bbce3c39e53b81dcf95430f780eb57911c
Jul 24 00:11:45.724084 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:45.724059    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e62089_0435_442b_b800_fb8a90646c90.slice/crio-778e8005c7395127ade3b5c346c95a10d2956642bb6a5a1d7bc589910603d56b WatchSource:0}: Error finding container 778e8005c7395127ade3b5c346c95a10d2956642bb6a5a1d7bc589910603d56b: Status 404 returned error can't find the container with id 778e8005c7395127ade3b5c346c95a10d2956642bb6a5a1d7bc589910603d56b
Jul 24 00:11:46.058477 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.058391    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j26n7" event={"ID":"a256a9ba-91cf-47ba-8e3b-a987081eac63","Type":"ContainerStarted","Data":"a3b8cb1864480b0e208cc3d4bbc319bbce3c39e53b81dcf95430f780eb57911c"}
Jul 24 00:11:46.059641 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.059614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-95fmv" event={"ID":"b3e62089-0435-442b-b800-fb8a90646c90","Type":"ContainerStarted","Data":"778e8005c7395127ade3b5c346c95a10d2956642bb6a5a1d7bc589910603d56b"}
Jul 24 00:11:46.061737 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.061712    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72lwd" event={"ID":"418b27c1-b425-4aed-a5e1-1f59b7a817b2","Type":"ContainerStarted","Data":"b348ccf865e9b5a9f1e6ef8f8b1f44a1c218da36bed60e5529b475387562fd43"}
Jul 24 00:11:46.063235 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.063212    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw" event={"ID":"cf5cfa3d-5b43-441d-b437-7741c5e87560","Type":"ContainerStarted","Data":"15363d94df15f96e78a8cd520004873aea5775ca97d2cbe20916903f862f31ea"}
Jul 24 00:11:46.063396 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.063381    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:46.065414 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.065393    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd" event={"ID":"bbc389d0-fa5c-4620-93cf-335b25bd9e2a","Type":"ContainerStarted","Data":"67d2750ae238f352fe7cc00f518a0c360457456a343e712a2d703315393bfb22"}
Jul 24 00:11:46.065523 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.065427    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd" event={"ID":"bbc389d0-fa5c-4620-93cf-335b25bd9e2a","Type":"ContainerStarted","Data":"3a77e2bc0b17819ffd9a1f654bc8755ac34984d6abce0bc6768ab33e11d9778b"}
Jul 24 00:11:46.065523 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.065478    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw"
Jul 24 00:11:46.065523 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.065519    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd" event={"ID":"bbc389d0-fa5c-4620-93cf-335b25bd9e2a","Type":"ContainerStarted","Data":"cd0ca41132b63cbb4bace9e08283d9c9bca36cd0e8dabc11c14a8f37f735c4e2"}
Jul 24 00:11:46.066813 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.066793    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm" event={"ID":"95829203-4e33-4406-8223-e860f545755b","Type":"ContainerStarted","Data":"35481f7d1cf273c76d25c9e442fc31080b46c641696e287f92e66ab1b6216689"}
Jul 24 00:11:46.068017 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.067998    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fmf4j" event={"ID":"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7","Type":"ContainerStarted","Data":"173a72c32c1547fbfa918c9a36332be1b2f1eca52d8a9fc2ec59c6093ac85008"}
Jul 24 00:11:46.087648 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.087602    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-72lwd" podStartSLOduration=16.827746728 podStartE2EDuration="33.087589295s" totalImagesPullingTime="16.259842567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:11:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:46.086007369 +0000 UTC m=+68.934326110" watchObservedRunningTime="2026-07-24 00:11:46.087589295 +0000 UTC m=+68.935908037"
Jul 24 00:11:46.113996 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.113958    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4f5c7bd6-q72gd" podStartSLOduration=1.57411391 podStartE2EDuration="11.113946009s" totalImagesPullingTime="9.539832099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:46.113042031 +0000 UTC m=+68.961360783" watchObservedRunningTime="2026-07-24 00:11:46.113946009 +0000 UTC m=+68.962264750"
Jul 24 00:11:46.129045 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.129008    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4bf64fd6-8mngm" podStartSLOduration=1.5757570680000001 podStartE2EDuration="11.128995719s" totalImagesPullingTime="9.553238651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:46.128670175 +0000 UTC m=+68.976988917" watchObservedRunningTime="2026-07-24 00:11:46.128995719 +0000 UTC m=+68.977314460"
Jul 24 00:11:46.146923 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:46.146885    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-747d9d96b4-4dkzw" podStartSLOduration=1.637006221 podStartE2EDuration="11.146873624s" totalImagesPullingTime="9.509867403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:46.146815288 +0000 UTC m=+68.995134028" watchObservedRunningTime="2026-07-24 00:11:46.146873624 +0000 UTC m=+68.995192366"
Jul 24 00:11:47.081209 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:47.081124    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fmf4j" event={"ID":"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7","Type":"ContainerStarted","Data":"9f797bc93942b0865c9ed549b2e7fba01612e0b69cb49f3a0c0c79390eee933a"}
Jul 24 00:11:48.088040 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:48.087758    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fmf4j" event={"ID":"1caa4b14-4b7d-43c5-b0c6-42a1b70e02f7","Type":"ContainerStarted","Data":"af4f8b03e05ed8d80390fe39a171c332c4a099bc4fd3bac739a627a98f0c4c27"}
Jul 24 00:11:48.105979 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:48.105928    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fmf4j" podStartSLOduration=69.86371147 podStartE2EDuration="1m11.105913476s" totalImagesPullingTime="1.242202006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:48.1047151 +0000 UTC m=+70.953033841" watchObservedRunningTime="2026-07-24 00:11:48.105913476 +0000 UTC m=+70.954232217"
Jul 24 00:11:51.100369 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:51.100328    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j26n7" event={"ID":"a256a9ba-91cf-47ba-8e3b-a987081eac63","Type":"ContainerStarted","Data":"304b58ddc1324253641c23ca3521869f3ed6638bd3d97b4fe9bcaaaafab43d25"}
Jul 24 00:11:51.101800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:51.101773    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-95fmv" event={"ID":"b3e62089-0435-442b-b800-fb8a90646c90","Type":"ContainerStarted","Data":"d86a39e6212dc75ae22fa6f02dcea94e7281dc6f200429abf6ea8a87df39f9f3"}
Jul 24 00:11:51.101932 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:51.101834    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:11:51.117636 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:51.117590    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j26n7" podStartSLOduration=69.417235176 podStartE2EDuration="1m14.117576576s" totalImagesPullingTime="4.7003414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:51.117478448 +0000 UTC m=+73.965797201" watchObservedRunningTime="2026-07-24 00:11:51.117576576 +0000 UTC m=+73.965895310"
Jul 24 00:11:51.136175 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:51.136138    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-95fmv" podStartSLOduration=69.439214905 podStartE2EDuration="1m14.13612689s" totalImagesPullingTime="4.696911985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:51.134993242 +0000 UTC m=+73.983311975" watchObservedRunningTime="2026-07-24 00:11:51.13612689 +0000 UTC m=+73.984445634"
Jul 24 00:11:55.404347 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:55.404300    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:11:58.028023 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:58.027993    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66ffc854c8-5t479"
Jul 24 00:11:59.226110 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.226073    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8vlz6"]
Jul 24 00:11:59.229415 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.229391    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.232610 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.232580    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 00:11:59.232916 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.232895    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 00:11:59.233114 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.233090    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pmp2f\""
Jul 24 00:11:59.233190 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.233148    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 00:11:59.233190 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.233180    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 00:11:59.233660 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.233639    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 00:11:59.234220 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.234205    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 00:11:59.321631 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.321592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.321631 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.321638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-wtmp\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.321895 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.321666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-tls\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.321895 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.321737    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-accelerators-collector-config\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.321895 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.321770    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-sys\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.321895 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.321818    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-metrics-client-ca\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.321895 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.321843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-textfile\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.321895 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.321887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-root\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.322179 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.321910    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49b98\" (UniqueName: \"kubernetes.io/projected/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-kube-api-access-49b98\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.422732 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.422700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-root\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.422732 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.422734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49b98\" (UniqueName: \"kubernetes.io/projected/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-kube-api-access-49b98\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.422961 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.422764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.422961 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.422790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-wtmp\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.422961 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.422804    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-root\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.422961 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.422821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-tls\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.422961 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.422867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-accelerators-collector-config\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.422961 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.422896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-sys\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.422961 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.422921    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-metrics-client-ca\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.422961 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.422946    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-textfile\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.423279 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.422994    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-sys\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.423279 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.423007    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-wtmp\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.423440 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.423421    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-textfile\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.423618 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.423591    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-accelerators-collector-config\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.423618 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.423611    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-metrics-client-ca\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.425175 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.425157    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-tls\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.425259 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.425244    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.432096 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.432077    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49b98\" (UniqueName: \"kubernetes.io/projected/a4c8622d-3eb6-4d2c-b71e-0866019ceebd-kube-api-access-49b98\") pod \"node-exporter-8vlz6\" (UID: \"a4c8622d-3eb6-4d2c-b71e-0866019ceebd\") " pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.541009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:11:59.540928    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8vlz6"
Jul 24 00:11:59.551999 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:11:59.551973    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c8622d_3eb6_4d2c_b71e_0866019ceebd.slice/crio-30d0d666539c67dd3f41c203b58093371ce3cda34872bc1fe41ee7f7dd71c8ae WatchSource:0}: Error finding container 30d0d666539c67dd3f41c203b58093371ce3cda34872bc1fe41ee7f7dd71c8ae: Status 404 returned error can't find the container with id 30d0d666539c67dd3f41c203b58093371ce3cda34872bc1fe41ee7f7dd71c8ae
Jul 24 00:12:00.134624 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.134581    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8vlz6" event={"ID":"a4c8622d-3eb6-4d2c-b71e-0866019ceebd","Type":"ContainerStarted","Data":"30d0d666539c67dd3f41c203b58093371ce3cda34872bc1fe41ee7f7dd71c8ae"}
Jul 24 00:12:00.423629 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.423532    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt" podUID="77373f87-2556-48c0-b398-50315b64a2be" containerName="registry" containerID="cri-o://d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87" gracePeriod=30
Jul 24 00:12:00.680714 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.680659    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:12:00.730768 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.730733    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqst7\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-kube-api-access-tqst7\") pod \"77373f87-2556-48c0-b398-50315b64a2be\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") "
Jul 24 00:12:00.730951 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.730781    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls\") pod \"77373f87-2556-48c0-b398-50315b64a2be\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") "
Jul 24 00:12:00.730951 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.730827    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-image-registry-private-configuration\") pod \"77373f87-2556-48c0-b398-50315b64a2be\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") "
Jul 24 00:12:00.730951 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.730867    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-trusted-ca\") pod \"77373f87-2556-48c0-b398-50315b64a2be\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") "
Jul 24 00:12:00.730951 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.730910    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-registry-certificates\") pod \"77373f87-2556-48c0-b398-50315b64a2be\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") "
Jul 24 00:12:00.731163 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.730958    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77373f87-2556-48c0-b398-50315b64a2be-ca-trust-extracted\") pod \"77373f87-2556-48c0-b398-50315b64a2be\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") "
Jul 24 00:12:00.731163 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.730999    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-installation-pull-secrets\") pod \"77373f87-2556-48c0-b398-50315b64a2be\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") "
Jul 24 00:12:00.731503 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.731462    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "77373f87-2556-48c0-b398-50315b64a2be" (UID: "77373f87-2556-48c0-b398-50315b64a2be"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:12:00.732009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.731977    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "77373f87-2556-48c0-b398-50315b64a2be" (UID: "77373f87-2556-48c0-b398-50315b64a2be"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:12:00.733584 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.733536    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-kube-api-access-tqst7" (OuterVolumeSpecName: "kube-api-access-tqst7") pod "77373f87-2556-48c0-b398-50315b64a2be" (UID: "77373f87-2556-48c0-b398-50315b64a2be"). InnerVolumeSpecName "kube-api-access-tqst7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:12:00.733680 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.733587    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "77373f87-2556-48c0-b398-50315b64a2be" (UID: "77373f87-2556-48c0-b398-50315b64a2be"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:12:00.733680 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.733638    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "77373f87-2556-48c0-b398-50315b64a2be" (UID: "77373f87-2556-48c0-b398-50315b64a2be"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:12:00.733816 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.733773    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "77373f87-2556-48c0-b398-50315b64a2be" (UID: "77373f87-2556-48c0-b398-50315b64a2be"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:12:00.740419 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.740388    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77373f87-2556-48c0-b398-50315b64a2be-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "77373f87-2556-48c0-b398-50315b64a2be" (UID: "77373f87-2556-48c0-b398-50315b64a2be"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:12:00.831594 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.831557    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-bound-sa-token\") pod \"77373f87-2556-48c0-b398-50315b64a2be\" (UID: \"77373f87-2556-48c0-b398-50315b64a2be\") "
Jul 24 00:12:00.831779 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.831738    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-registry-certificates\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:12:00.831779 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.831759    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77373f87-2556-48c0-b398-50315b64a2be-ca-trust-extracted\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:12:00.831779 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.831773    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-installation-pull-secrets\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:12:00.831934 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.831785    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqst7\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-kube-api-access-tqst7\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:12:00.831934 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.831798    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-registry-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:12:00.831934 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.831812    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77373f87-2556-48c0-b398-50315b64a2be-image-registry-private-configuration\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:12:00.831934 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.831834    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77373f87-2556-48c0-b398-50315b64a2be-trusted-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:12:00.833765 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.833738    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "77373f87-2556-48c0-b398-50315b64a2be" (UID: "77373f87-2556-48c0-b398-50315b64a2be"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:12:00.932260 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:00.932173    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77373f87-2556-48c0-b398-50315b64a2be-bound-sa-token\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:12:01.139790 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:01.139758    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8vlz6" event={"ID":"a4c8622d-3eb6-4d2c-b71e-0866019ceebd","Type":"ContainerStarted","Data":"736d853b8241c1b9dde486ea3dd77d71ce6e8eba31fb9c5c99f487f292fc7240"}
Jul 24 00:12:01.141130 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:01.141097    2574 generic.go:356] "Generic (PLEG): container finished" podID="77373f87-2556-48c0-b398-50315b64a2be" containerID="d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87" exitCode=0
Jul 24 00:12:01.141287 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:01.141137    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt"
Jul 24 00:12:01.141287 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:01.141181    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt" event={"ID":"77373f87-2556-48c0-b398-50315b64a2be","Type":"ContainerDied","Data":"d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87"}
Jul 24 00:12:01.141287 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:01.141233    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546978b9f4-lqbqt" event={"ID":"77373f87-2556-48c0-b398-50315b64a2be","Type":"ContainerDied","Data":"f77764b7d97a197f44df5be1610312c402b584263536c4bb11a00c8317dcd6dc"}
Jul 24 00:12:01.141287 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:01.141256    2574 scope.go:117] "RemoveContainer" containerID="d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87"
Jul 24 00:12:01.150440 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:01.150423    2574 scope.go:117] "RemoveContainer" containerID="d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87"
Jul 24 00:12:01.150754 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:12:01.150727    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87\": container with ID starting with d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87 not found: ID does not exist" containerID="d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87"
Jul 24 00:12:01.150849 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:01.150806    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87"} err="failed to get container status \"d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87\": rpc error: code = NotFound desc = could not find container \"d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87\": container with ID starting with d5b12544d6ce3a579186e969118ea039740f1482579c68139a41e6d6f136ca87 not found: ID does not exist"
Jul 24 00:12:01.203575 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:01.203542    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-546978b9f4-lqbqt"]
Jul 24 00:12:01.207917 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:01.207893    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-546978b9f4-lqbqt"]
Jul 24 00:12:01.750562 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:01.750528    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77373f87-2556-48c0-b398-50315b64a2be" path="/var/lib/kubelet/pods/77373f87-2556-48c0-b398-50315b64a2be/volumes"
Jul 24 00:12:02.147548 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:02.147515    2574 generic.go:356] "Generic (PLEG): container finished" podID="a4c8622d-3eb6-4d2c-b71e-0866019ceebd" containerID="736d853b8241c1b9dde486ea3dd77d71ce6e8eba31fb9c5c99f487f292fc7240" exitCode=0
Jul 24 00:12:02.147752 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:02.147560    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8vlz6" event={"ID":"a4c8622d-3eb6-4d2c-b71e-0866019ceebd","Type":"ContainerDied","Data":"736d853b8241c1b9dde486ea3dd77d71ce6e8eba31fb9c5c99f487f292fc7240"}
Jul 24 00:12:03.153082 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:03.153041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8vlz6" event={"ID":"a4c8622d-3eb6-4d2c-b71e-0866019ceebd","Type":"ContainerStarted","Data":"0edced4979b9830468a6bb6786e1cd09c2e055adeff975cf8417bc3032c05630"}
Jul 24 00:12:03.153470 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:03.153098    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8vlz6" event={"ID":"a4c8622d-3eb6-4d2c-b71e-0866019ceebd","Type":"ContainerStarted","Data":"f8c4cfaf7edac07b1a9e4d690c47ef1ace1fa15950273d8316ad261ef38c8b4c"}
Jul 24 00:12:22.107821 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:22.107790    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-95fmv"
Jul 24 00:12:22.123113 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:12:22.123059    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8vlz6" podStartSLOduration=21.586586099 podStartE2EDuration="23.123041291s" totalImagesPullingTime="1.471755701s" totalInitContainerRuntime="64.699491ms" isStatefulPod=true podCreationTimestamp="2026-07-24 00:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:12:03.195879346 +0000 UTC m=+86.044198086" watchObservedRunningTime="2026-07-24 00:12:22.123041291 +0000 UTC m=+104.971360027"
Jul 24 00:15:37.646164 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:15:37.646137    2574 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 00:16:11.678551 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.678515    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"]
Jul 24 00:16:11.680160 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.678742    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77373f87-2556-48c0-b398-50315b64a2be" containerName="registry"
Jul 24 00:16:11.680160 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.678752    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="77373f87-2556-48c0-b398-50315b64a2be" containerName="registry"
Jul 24 00:16:11.680160 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.678802    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="77373f87-2556-48c0-b398-50315b64a2be" containerName="registry"
Jul 24 00:16:11.680658 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.680644    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.683191 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.683168    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 00:16:11.683454 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.683437    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g5wcl\""
Jul 24 00:16:11.683557 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.683544    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 00:16:11.683608 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.683594    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 00:16:11.683802 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.683787    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 00:16:11.701478 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.701457    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"]
Jul 24 00:16:11.818247 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.818218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2583d812-d620-4302-a7ae-d9aa54fc30fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.818467 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.818283    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2583d812-d620-4302-a7ae-d9aa54fc30fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.818467 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.818331    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2583d812-d620-4302-a7ae-d9aa54fc30fc-manifests\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.818467 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.818425    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2583d812-d620-4302-a7ae-d9aa54fc30fc-tmp\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.818467 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.818452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6fc\" (UniqueName: \"kubernetes.io/projected/2583d812-d620-4302-a7ae-d9aa54fc30fc-kube-api-access-dk6fc\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.919791 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.919761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6fc\" (UniqueName: \"kubernetes.io/projected/2583d812-d620-4302-a7ae-d9aa54fc30fc-kube-api-access-dk6fc\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.919965 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.919807    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2583d812-d620-4302-a7ae-d9aa54fc30fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.919965 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.919840    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2583d812-d620-4302-a7ae-d9aa54fc30fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.919965 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.919856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2583d812-d620-4302-a7ae-d9aa54fc30fc-manifests\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.919965 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.919874    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2583d812-d620-4302-a7ae-d9aa54fc30fc-tmp\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.920354 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.920308    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2583d812-d620-4302-a7ae-d9aa54fc30fc-manifests\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.922200 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.922176    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2583d812-d620-4302-a7ae-d9aa54fc30fc-tmp\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.922285 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.922235    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2583d812-d620-4302-a7ae-d9aa54fc30fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.922367 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.922358    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2583d812-d620-4302-a7ae-d9aa54fc30fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.927619 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.927588    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6fc\" (UniqueName: \"kubernetes.io/projected/2583d812-d620-4302-a7ae-d9aa54fc30fc-kube-api-access-dk6fc\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-ssxwz\" (UID: \"2583d812-d620-4302-a7ae-d9aa54fc30fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:11.990895 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:11.990815    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:12.120975 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:12.120942    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"]
Jul 24 00:16:12.124203 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:16:12.124179    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2583d812_d620_4302_a7ae_d9aa54fc30fc.slice/crio-16560fbe60f722138755f222cde25cb884430895a52f6be92c907a8d73cec883 WatchSource:0}: Error finding container 16560fbe60f722138755f222cde25cb884430895a52f6be92c907a8d73cec883: Status 404 returned error can't find the container with id 16560fbe60f722138755f222cde25cb884430895a52f6be92c907a8d73cec883
Jul 24 00:16:12.125827 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:12.125810    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:16:12.949969 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:12.949923    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz" event={"ID":"2583d812-d620-4302-a7ae-d9aa54fc30fc","Type":"ContainerStarted","Data":"16560fbe60f722138755f222cde25cb884430895a52f6be92c907a8d73cec883"}
Jul 24 00:16:15.962662 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:15.962628    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz" event={"ID":"2583d812-d620-4302-a7ae-d9aa54fc30fc","Type":"ContainerStarted","Data":"bb913e581c25c934db7b3fb8faece3dad2af16f36a554d369c866c7b8f8551ec"}
Jul 24 00:16:16.967570 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:16.967532    2574 generic.go:356] "Generic (PLEG): container finished" podID="2583d812-d620-4302-a7ae-d9aa54fc30fc" containerID="bb913e581c25c934db7b3fb8faece3dad2af16f36a554d369c866c7b8f8551ec" exitCode=0
Jul 24 00:16:16.967955 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:16.967618    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz" event={"ID":"2583d812-d620-4302-a7ae-d9aa54fc30fc","Type":"ContainerDied","Data":"bb913e581c25c934db7b3fb8faece3dad2af16f36a554d369c866c7b8f8551ec"}
Jul 24 00:16:17.972632 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:17.972598    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz" event={"ID":"2583d812-d620-4302-a7ae-d9aa54fc30fc","Type":"ContainerStarted","Data":"7c6c39ccd1eed9f69bb8a1d3f351676cfe7a1e2e4f8c27bebd724d16af24149e"}
Jul 24 00:16:17.973016 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:17.972733    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:18.012082 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:18.012007    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz" podStartSLOduration=2.692090699 podStartE2EDuration="7.011990574s" totalImagesPullingTime="3.313553935s" totalInitContainerRuntime="1.00634594s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:18.010940127 +0000 UTC m=+340.859258852" watchObservedRunningTime="2026-07-24 00:16:18.011990574 +0000 UTC m=+340.860309316"
Jul 24 00:16:28.977654 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:28.977624    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-ssxwz"
Jul 24 00:16:31.361597 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.361522    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"]
Jul 24 00:16:31.364738 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.364716    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"
Jul 24 00:16:31.375827 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.375801    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 00:16:31.375934 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.375841    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 00:16:31.376151 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.376133    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fkk28\""
Jul 24 00:16:31.376873 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.376858    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 00:16:31.378193 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.378177    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 00:16:31.410107 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.410082    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"]
Jul 24 00:16:31.465278 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.465248    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d70df8-bf64-4fb9-88e0-b9955f896137-tls-certs\") pod \"kube-auth-proxy-57dc97f664-qqqdd\" (UID: \"49d70df8-bf64-4fb9-88e0-b9955f896137\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"
Jul 24 00:16:31.465420 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.465289    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d70df8-bf64-4fb9-88e0-b9955f896137-tmp\") pod \"kube-auth-proxy-57dc97f664-qqqdd\" (UID: \"49d70df8-bf64-4fb9-88e0-b9955f896137\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"
Jul 24 00:16:31.465420 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.465389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvxk9\" (UniqueName: \"kubernetes.io/projected/49d70df8-bf64-4fb9-88e0-b9955f896137-kube-api-access-dvxk9\") pod \"kube-auth-proxy-57dc97f664-qqqdd\" (UID: \"49d70df8-bf64-4fb9-88e0-b9955f896137\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"
Jul 24 00:16:31.566224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.566188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvxk9\" (UniqueName: \"kubernetes.io/projected/49d70df8-bf64-4fb9-88e0-b9955f896137-kube-api-access-dvxk9\") pod \"kube-auth-proxy-57dc97f664-qqqdd\" (UID: \"49d70df8-bf64-4fb9-88e0-b9955f896137\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"
Jul 24 00:16:31.566412 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.566247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d70df8-bf64-4fb9-88e0-b9955f896137-tls-certs\") pod \"kube-auth-proxy-57dc97f664-qqqdd\" (UID: \"49d70df8-bf64-4fb9-88e0-b9955f896137\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"
Jul 24 00:16:31.566412 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.566278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d70df8-bf64-4fb9-88e0-b9955f896137-tmp\") pod \"kube-auth-proxy-57dc97f664-qqqdd\" (UID: \"49d70df8-bf64-4fb9-88e0-b9955f896137\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"
Jul 24 00:16:31.568588 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.568565    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d70df8-bf64-4fb9-88e0-b9955f896137-tmp\") pod \"kube-auth-proxy-57dc97f664-qqqdd\" (UID: \"49d70df8-bf64-4fb9-88e0-b9955f896137\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"
Jul 24 00:16:31.568905 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.568884    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d70df8-bf64-4fb9-88e0-b9955f896137-tls-certs\") pod \"kube-auth-proxy-57dc97f664-qqqdd\" (UID: \"49d70df8-bf64-4fb9-88e0-b9955f896137\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"
Jul 24 00:16:31.579202 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.579180    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvxk9\" (UniqueName: \"kubernetes.io/projected/49d70df8-bf64-4fb9-88e0-b9955f896137-kube-api-access-dvxk9\") pod \"kube-auth-proxy-57dc97f664-qqqdd\" (UID: \"49d70df8-bf64-4fb9-88e0-b9955f896137\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"
Jul 24 00:16:31.674776 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.674704    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"
Jul 24 00:16:31.809200 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:31.809166    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd"]
Jul 24 00:16:31.812361 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:16:31.812325    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d70df8_bf64_4fb9_88e0_b9955f896137.slice/crio-d01f656432c285d3bd87ff77656208f7e9a3df2c194b80b53690bb342e033b67 WatchSource:0}: Error finding container d01f656432c285d3bd87ff77656208f7e9a3df2c194b80b53690bb342e033b67: Status 404 returned error can't find the container with id d01f656432c285d3bd87ff77656208f7e9a3df2c194b80b53690bb342e033b67
Jul 24 00:16:32.018992 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:32.018909    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd" event={"ID":"49d70df8-bf64-4fb9-88e0-b9955f896137","Type":"ContainerStarted","Data":"d01f656432c285d3bd87ff77656208f7e9a3df2c194b80b53690bb342e033b67"}
Jul 24 00:16:35.031778 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:35.031693    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd" event={"ID":"49d70df8-bf64-4fb9-88e0-b9955f896137","Type":"ContainerStarted","Data":"5ddc3b5f037d29a4408c919c9ce36b8122de00a658ee1e86b4205bdd797d4af0"}
Jul 24 00:16:35.052656 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:35.052608    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57dc97f664-qqqdd" podStartSLOduration=1.248892581 podStartE2EDuration="4.052592548s" totalImagesPullingTime="2.803699967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:35.051206146 +0000 UTC m=+357.899524895" watchObservedRunningTime="2026-07-24 00:16:35.052592548 +0000 UTC m=+357.900911290"
Jul 24 00:16:38.991456 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:38.991427    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"]
Jul 24 00:16:38.999554 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:38.999529    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.002360 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.002338    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-twg8t\""
Jul 24 00:16:39.002680 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.002664    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 00:16:39.023738 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.023711    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.023845 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.023737    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.023845 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.023768    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352617c6-6d55-4cb9-9db8-53f142113c63-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.023845 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.023822    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.024008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.023870    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.024008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.023901    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.024008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.023931    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.024008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.023961    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352617c6-6d55-4cb9-9db8-53f142113c63-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.024008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.023994    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zcck\" (UniqueName: \"kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-kube-api-access-8zcck\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.037424 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.037404    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"]
Jul 24 00:16:39.112736 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.112703    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"]
Jul 24 00:16:39.115706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.115683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.119245 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.119213    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 00:16:39.119375 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.119331    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 00:16:39.119375 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.119340    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 00:16:39.119484 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.119348    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-prv66\""
Jul 24 00:16:39.119732 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.119717    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 00:16:39.119893 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.119876    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 00:16:39.124277 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124259    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352617c6-6d55-4cb9-9db8-53f142113c63-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.124413 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcck\" (UniqueName: \"kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-kube-api-access-8zcck\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.124413 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124308    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9mh\" (UniqueName: \"kubernetes.io/projected/ae5e41e2-699c-42ce-91ef-453faac576fc-kube-api-access-pz9mh\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.124413 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124398    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.124558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124440    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.124558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124474    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352617c6-6d55-4cb9-9db8-53f142113c63-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.124558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae5e41e2-699c-42ce-91ef-453faac576fc-manager-config\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.124558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124547    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae5e41e2-699c-42ce-91ef-453faac576fc-cert\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.124811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124671    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.124811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124711    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.124811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124742    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.124811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.124811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124795    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.124811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.124802    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae5e41e2-699c-42ce-91ef-453faac576fc-metrics-cert\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.125165 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.125143    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.125221 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.125177    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.125275 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.125240    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352617c6-6d55-4cb9-9db8-53f142113c63-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.125398 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.125377    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.127047 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.127024    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352617c6-6d55-4cb9-9db8-53f142113c63-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.127115 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.127049    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.134554 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.134536    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"]
Jul 24 00:16:39.142144 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.142121    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.142361 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.142346    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcck\" (UniqueName: \"kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-kube-api-access-8zcck\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.225270 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.225231    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae5e41e2-699c-42ce-91ef-453faac576fc-manager-config\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.225473 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.225279    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae5e41e2-699c-42ce-91ef-453faac576fc-cert\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.225473 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.225369    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae5e41e2-699c-42ce-91ef-453faac576fc-metrics-cert\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.225473 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.225424    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9mh\" (UniqueName: \"kubernetes.io/projected/ae5e41e2-699c-42ce-91ef-453faac576fc-kube-api-access-pz9mh\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.226024 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.226001    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae5e41e2-699c-42ce-91ef-453faac576fc-manager-config\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.228040 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.227994    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae5e41e2-699c-42ce-91ef-453faac576fc-metrics-cert\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.228153 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.228057    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae5e41e2-699c-42ce-91ef-453faac576fc-cert\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.243634 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.243576    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9mh\" (UniqueName: \"kubernetes.io/projected/ae5e41e2-699c-42ce-91ef-453faac576fc-kube-api-access-pz9mh\") pod \"lws-controller-manager-5bbf775ccb-rh6mb\" (UID: \"ae5e41e2-699c-42ce-91ef-453faac576fc\") " pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.309543 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.309513    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:39.426800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.426773    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:39.459864 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.459837    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"]
Jul 24 00:16:39.460769 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:16:39.460741    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352617c6_6d55_4cb9_9db8_53f142113c63.slice/crio-028f813aba5196a977e5bc11213c5f8b15ce43e5eba11e5b502d9ed9c44eea21 WatchSource:0}: Error finding container 028f813aba5196a977e5bc11213c5f8b15ce43e5eba11e5b502d9ed9c44eea21: Status 404 returned error can't find the container with id 028f813aba5196a977e5bc11213c5f8b15ce43e5eba11e5b502d9ed9c44eea21
Jul 24 00:16:39.566046 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:39.566020    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"]
Jul 24 00:16:39.568773 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:16:39.568749    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5e41e2_699c_42ce_91ef_453faac576fc.slice/crio-c2618314ef7b1fd92f1d161c6f9d2503a9ffcf6776c63d69b6bdc39feda857f4 WatchSource:0}: Error finding container c2618314ef7b1fd92f1d161c6f9d2503a9ffcf6776c63d69b6bdc39feda857f4: Status 404 returned error can't find the container with id c2618314ef7b1fd92f1d161c6f9d2503a9ffcf6776c63d69b6bdc39feda857f4
Jul 24 00:16:40.048483 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:40.048450    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb" event={"ID":"ae5e41e2-699c-42ce-91ef-453faac576fc","Type":"ContainerStarted","Data":"c2618314ef7b1fd92f1d161c6f9d2503a9ffcf6776c63d69b6bdc39feda857f4"}
Jul 24 00:16:40.049574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:40.049551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t" event={"ID":"352617c6-6d55-4cb9-9db8-53f142113c63","Type":"ContainerStarted","Data":"028f813aba5196a977e5bc11213c5f8b15ce43e5eba11e5b502d9ed9c44eea21"}
Jul 24 00:16:44.278794 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:44.278709    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"]
Jul 24 00:16:45.069024 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:45.068990    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb" event={"ID":"ae5e41e2-699c-42ce-91ef-453faac576fc","Type":"ContainerStarted","Data":"7274b084a8b02a2142aed66f5ad208944690f15503f23fd5f91f2af628ef4522"}
Jul 24 00:16:45.069191 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:45.069060    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:16:45.090346 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:45.090278    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb" podStartSLOduration=1.6490416620000001 podStartE2EDuration="6.090263158s" totalImagesPullingTime="4.441221496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:45.08954553 +0000 UTC m=+367.937864265" watchObservedRunningTime="2026-07-24 00:16:45.090263158 +0000 UTC m=+367.938581899"
Jul 24 00:16:54.025918 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.025883    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:16:54.026161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.025961    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:16:54.026161 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.025990    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:16:54.101262 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.101225    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t" event={"ID":"352617c6-6d55-4cb9-9db8-53f142113c63","Type":"ContainerStarted","Data":"fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef"}
Jul 24 00:16:54.101414 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.101308    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t" podUID="352617c6-6d55-4cb9-9db8-53f142113c63" containerName="istio-proxy" containerID="cri-o://fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef" gracePeriod=30
Jul 24 00:16:54.122152 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.122108    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t" podStartSLOduration=1.559260016 podStartE2EDuration="16.122094312s" totalImagesPullingTime="14.562834296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:54.120047033 +0000 UTC m=+376.968365777" watchObservedRunningTime="2026-07-24 00:16:54.122094312 +0000 UTC m=+376.970413052"
Jul 24 00:16:54.310698 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.310628    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:54.333262 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.333239    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:54.444683 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.444641    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zcck\" (UniqueName: \"kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-kube-api-access-8zcck\") pod \"352617c6-6d55-4cb9-9db8-53f142113c63\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") "
Jul 24 00:16:54.444683 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.444699    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-certs\") pod \"352617c6-6d55-4cb9-9db8-53f142113c63\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") "
Jul 24 00:16:54.444933 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.444732    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-envoy\") pod \"352617c6-6d55-4cb9-9db8-53f142113c63\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") "
Jul 24 00:16:54.444933 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.444753    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352617c6-6d55-4cb9-9db8-53f142113c63-istio-podinfo\") pod \"352617c6-6d55-4cb9-9db8-53f142113c63\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") "
Jul 24 00:16:54.444933 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.444796    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-socket\") pod \"352617c6-6d55-4cb9-9db8-53f142113c63\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") "
Jul 24 00:16:54.444933 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.444819    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-credential-socket\") pod \"352617c6-6d55-4cb9-9db8-53f142113c63\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") "
Jul 24 00:16:54.444933 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.444881    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352617c6-6d55-4cb9-9db8-53f142113c63-istiod-ca-cert\") pod \"352617c6-6d55-4cb9-9db8-53f142113c63\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") "
Jul 24 00:16:54.444933 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.444927    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-istio-token\") pod \"352617c6-6d55-4cb9-9db8-53f142113c63\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") "
Jul 24 00:16:54.445221 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.444956    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-data\") pod \"352617c6-6d55-4cb9-9db8-53f142113c63\" (UID: \"352617c6-6d55-4cb9-9db8-53f142113c63\") "
Jul 24 00:16:54.445221 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.445018    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "352617c6-6d55-4cb9-9db8-53f142113c63" (UID: "352617c6-6d55-4cb9-9db8-53f142113c63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:16:54.445221 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.445127    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "352617c6-6d55-4cb9-9db8-53f142113c63" (UID: "352617c6-6d55-4cb9-9db8-53f142113c63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:16:54.445386 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.445240    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-data" (OuterVolumeSpecName: "istio-data") pod "352617c6-6d55-4cb9-9db8-53f142113c63" (UID: "352617c6-6d55-4cb9-9db8-53f142113c63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:16:54.445386 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.445293    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:16:54.445386 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.445341    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-workload-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:16:54.445524 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.445399    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352617c6-6d55-4cb9-9db8-53f142113c63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "352617c6-6d55-4cb9-9db8-53f142113c63" (UID: "352617c6-6d55-4cb9-9db8-53f142113c63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:16:54.445524 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.445408    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "352617c6-6d55-4cb9-9db8-53f142113c63" (UID: "352617c6-6d55-4cb9-9db8-53f142113c63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:16:54.447078 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.447048    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-kube-api-access-8zcck" (OuterVolumeSpecName: "kube-api-access-8zcck") pod "352617c6-6d55-4cb9-9db8-53f142113c63" (UID: "352617c6-6d55-4cb9-9db8-53f142113c63"). InnerVolumeSpecName "kube-api-access-8zcck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:16:54.447213 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.447094    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-istio-token" (OuterVolumeSpecName: "istio-token") pod "352617c6-6d55-4cb9-9db8-53f142113c63" (UID: "352617c6-6d55-4cb9-9db8-53f142113c63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:16:54.447335 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.447299    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "352617c6-6d55-4cb9-9db8-53f142113c63" (UID: "352617c6-6d55-4cb9-9db8-53f142113c63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:16:54.447400 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.447340    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/352617c6-6d55-4cb9-9db8-53f142113c63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "352617c6-6d55-4cb9-9db8-53f142113c63" (UID: "352617c6-6d55-4cb9-9db8-53f142113c63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 00:16:54.546549 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.546521    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-envoy\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:16:54.546549 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.546545    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352617c6-6d55-4cb9-9db8-53f142113c63-istio-podinfo\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:16:54.546549 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.546554    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-credential-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:16:54.546754 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.546562    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352617c6-6d55-4cb9-9db8-53f142113c63-istiod-ca-cert\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:16:54.546754 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.546573    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-istio-token\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:16:54.546754 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.546580    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352617c6-6d55-4cb9-9db8-53f142113c63-istio-data\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:16:54.546754 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:54.546587    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zcck\" (UniqueName: \"kubernetes.io/projected/352617c6-6d55-4cb9-9db8-53f142113c63-kube-api-access-8zcck\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:16:55.106019 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:55.105990    2574 generic.go:356] "Generic (PLEG): container finished" podID="352617c6-6d55-4cb9-9db8-53f142113c63" containerID="fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef" exitCode=2
Jul 24 00:16:55.106487 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:55.106034    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"
Jul 24 00:16:55.106487 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:55.106055    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t" event={"ID":"352617c6-6d55-4cb9-9db8-53f142113c63","Type":"ContainerDied","Data":"fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef"}
Jul 24 00:16:55.106487 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:55.106102    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t" event={"ID":"352617c6-6d55-4cb9-9db8-53f142113c63","Type":"ContainerDied","Data":"028f813aba5196a977e5bc11213c5f8b15ce43e5eba11e5b502d9ed9c44eea21"}
Jul 24 00:16:55.106487 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:55.106125    2574 scope.go:117] "RemoveContainer" containerID="fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef"
Jul 24 00:16:55.114177 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:55.114160    2574 scope.go:117] "RemoveContainer" containerID="fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef"
Jul 24 00:16:55.114435 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:16:55.114417    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef\": container with ID starting with fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef not found: ID does not exist" containerID="fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef"
Jul 24 00:16:55.114496 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:55.114443    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef"} err="failed to get container status \"fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef\": rpc error: code = NotFound desc = could not find container \"fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef\": container with ID starting with fe14919a10a50ae4dfc71a86f09700935c2287e94ca56b57486e1325e68235ef not found: ID does not exist"
Jul 24 00:16:55.126191 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:55.126167    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"]
Jul 24 00:16:55.129459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:55.129440    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc9hj8t"]
Jul 24 00:16:55.751291 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:55.751255    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="352617c6-6d55-4cb9-9db8-53f142113c63" path="/var/lib/kubelet/pods/352617c6-6d55-4cb9-9db8-53f142113c63/volumes"
Jul 24 00:16:56.074788 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:16:56.074763    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5bbf775ccb-rh6mb"
Jul 24 00:19:13.767615 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.767582    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"]
Jul 24 00:19:13.768053 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.767928    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="352617c6-6d55-4cb9-9db8-53f142113c63" containerName="istio-proxy"
Jul 24 00:19:13.768053 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.767942    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="352617c6-6d55-4cb9-9db8-53f142113c63" containerName="istio-proxy"
Jul 24 00:19:13.768053 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.768015    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="352617c6-6d55-4cb9-9db8-53f142113c63" containerName="istio-proxy"
Jul 24 00:19:13.770150 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.770131    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.772620 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.772596    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 00:19:13.772884 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.772866    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h4qwd\""
Jul 24 00:19:13.780514 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.780492    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"]
Jul 24 00:19:13.871586 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.871544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.871586 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.871586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.871779 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.871606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.871779 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.871680    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.871779 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.871713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.871779 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.871755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sbkh\" (UniqueName: \"kubernetes.io/projected/a88fbbbb-bb27-4049-99b0-31b75cb659ed-kube-api-access-7sbkh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.871901 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.871784    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.871901 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.871804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.871901 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.871850    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.972855 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.972819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.972855 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.972855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973040 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.972878    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973040 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.973016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973138 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.973068    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973138 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.973126    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sbkh\" (UniqueName: \"kubernetes.io/projected/a88fbbbb-bb27-4049-99b0-31b75cb659ed-kube-api-access-7sbkh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973241 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.973160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973241 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.973186    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973241 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.973231    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973414 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.973256    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973414 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.973275    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973414 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.973385    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.973493    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.973706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.973684    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.975482 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.975456    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.975670 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.975654    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.981118 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.981092    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a88fbbbb-bb27-4049-99b0-31b75cb659ed-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:13.982120 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:13.982096    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sbkh\" (UniqueName: \"kubernetes.io/projected/a88fbbbb-bb27-4049-99b0-31b75cb659ed-kube-api-access-7sbkh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mfq7p\" (UID: \"a88fbbbb-bb27-4049-99b0-31b75cb659ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:14.080733 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:14.080701    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:14.208680 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:14.208597    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"]
Jul 24 00:19:14.210929 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:19:14.210892    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88fbbbb_bb27_4049_99b0_31b75cb659ed.slice/crio-bba408bdf0fecb4a7b2a1fd8e5227721ac808eda2d16bed46e06d8879121084f WatchSource:0}: Error finding container bba408bdf0fecb4a7b2a1fd8e5227721ac808eda2d16bed46e06d8879121084f: Status 404 returned error can't find the container with id bba408bdf0fecb4a7b2a1fd8e5227721ac808eda2d16bed46e06d8879121084f
Jul 24 00:19:14.212970 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:14.212939    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:19:14.213041 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:14.213017    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:19:14.213081 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:14.213045    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:19:14.593296 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:14.593262    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p" event={"ID":"a88fbbbb-bb27-4049-99b0-31b75cb659ed","Type":"ContainerStarted","Data":"72ebb2f1051aa5d23408b659e41b87bff5f77f6a69417eca5c28965486fdd87b"}
Jul 24 00:19:14.593493 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:14.593330    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p" event={"ID":"a88fbbbb-bb27-4049-99b0-31b75cb659ed","Type":"ContainerStarted","Data":"bba408bdf0fecb4a7b2a1fd8e5227721ac808eda2d16bed46e06d8879121084f"}
Jul 24 00:19:14.614727 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:14.614674    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p" podStartSLOduration=1.614659751 podStartE2EDuration="1.614659751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:14.612923241 +0000 UTC m=+517.461241985" watchObservedRunningTime="2026-07-24 00:19:14.614659751 +0000 UTC m=+517.462978492"
Jul 24 00:19:15.080909 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:15.080880    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:15.086046 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:15.086022    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:15.597362 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:15.597337    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:15.598207 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:15.598191    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:19:26.479704 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.479667    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-djndv"]
Jul 24 00:19:26.482566 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.482550    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-djndv"
Jul 24 00:19:26.485015 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.484992    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 00:19:26.486063 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.486039    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6qs5n\""
Jul 24 00:19:26.486175 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.486089    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 00:19:26.489858 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.489837    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-djndv"]
Jul 24 00:19:26.492768 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.492748    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-x8clm"]
Jul 24 00:19:26.495366 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.495351    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x8clm"
Jul 24 00:19:26.502595 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.502574    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x8clm"]
Jul 24 00:19:26.565404 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.565375    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ssm\" (UniqueName: \"kubernetes.io/projected/e574c8d5-1312-4639-b4c1-a149ab637f52-kube-api-access-p8ssm\") pod \"authorino-75d556bd77-x8clm\" (UID: \"e574c8d5-1312-4639-b4c1-a149ab637f52\") " pod="kuadrant-system/authorino-75d556bd77-x8clm"
Jul 24 00:19:26.666597 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.666563    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ssm\" (UniqueName: \"kubernetes.io/projected/e574c8d5-1312-4639-b4c1-a149ab637f52-kube-api-access-p8ssm\") pod \"authorino-75d556bd77-x8clm\" (UID: \"e574c8d5-1312-4639-b4c1-a149ab637f52\") " pod="kuadrant-system/authorino-75d556bd77-x8clm"
Jul 24 00:19:26.666774 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.666608    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9j9r\" (UniqueName: \"kubernetes.io/projected/6699209e-9d46-4132-ae42-8bb20e8c0a51-kube-api-access-s9j9r\") pod \"authorino-58cd955675-djndv\" (UID: \"6699209e-9d46-4132-ae42-8bb20e8c0a51\") " pod="kuadrant-system/authorino-58cd955675-djndv"
Jul 24 00:19:26.674252 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.674223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ssm\" (UniqueName: \"kubernetes.io/projected/e574c8d5-1312-4639-b4c1-a149ab637f52-kube-api-access-p8ssm\") pod \"authorino-75d556bd77-x8clm\" (UID: \"e574c8d5-1312-4639-b4c1-a149ab637f52\") " pod="kuadrant-system/authorino-75d556bd77-x8clm"
Jul 24 00:19:26.767270 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.767195    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9j9r\" (UniqueName: \"kubernetes.io/projected/6699209e-9d46-4132-ae42-8bb20e8c0a51-kube-api-access-s9j9r\") pod \"authorino-58cd955675-djndv\" (UID: \"6699209e-9d46-4132-ae42-8bb20e8c0a51\") " pod="kuadrant-system/authorino-58cd955675-djndv"
Jul 24 00:19:26.774912 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.774871    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9j9r\" (UniqueName: \"kubernetes.io/projected/6699209e-9d46-4132-ae42-8bb20e8c0a51-kube-api-access-s9j9r\") pod \"authorino-58cd955675-djndv\" (UID: \"6699209e-9d46-4132-ae42-8bb20e8c0a51\") " pod="kuadrant-system/authorino-58cd955675-djndv"
Jul 24 00:19:26.792586 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.792567    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-djndv"
Jul 24 00:19:26.804411 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.804387    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x8clm"
Jul 24 00:19:26.929017 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.928969    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-djndv"]
Jul 24 00:19:26.931595 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:19:26.931568    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6699209e_9d46_4132_ae42_8bb20e8c0a51.slice/crio-3b3d01f75289f7dccf9d17705d88a30f3c3e7f539673c287073dcf0a0af9fe69 WatchSource:0}: Error finding container 3b3d01f75289f7dccf9d17705d88a30f3c3e7f539673c287073dcf0a0af9fe69: Status 404 returned error can't find the container with id 3b3d01f75289f7dccf9d17705d88a30f3c3e7f539673c287073dcf0a0af9fe69
Jul 24 00:19:26.945493 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:26.945468    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x8clm"]
Jul 24 00:19:26.948236 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:19:26.948214    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode574c8d5_1312_4639_b4c1_a149ab637f52.slice/crio-99a7ea75bb78b7184d0ab979e3d1acb8c277cc77c881cfdb4e90b403e8cb79cf WatchSource:0}: Error finding container 99a7ea75bb78b7184d0ab979e3d1acb8c277cc77c881cfdb4e90b403e8cb79cf: Status 404 returned error can't find the container with id 99a7ea75bb78b7184d0ab979e3d1acb8c277cc77c881cfdb4e90b403e8cb79cf
Jul 24 00:19:27.642329 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:27.642238    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x8clm" event={"ID":"e574c8d5-1312-4639-b4c1-a149ab637f52","Type":"ContainerStarted","Data":"99a7ea75bb78b7184d0ab979e3d1acb8c277cc77c881cfdb4e90b403e8cb79cf"}
Jul 24 00:19:27.644802 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:27.644746    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-djndv" event={"ID":"6699209e-9d46-4132-ae42-8bb20e8c0a51","Type":"ContainerStarted","Data":"3b3d01f75289f7dccf9d17705d88a30f3c3e7f539673c287073dcf0a0af9fe69"}
Jul 24 00:19:30.659479 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:30.659383    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x8clm" event={"ID":"e574c8d5-1312-4639-b4c1-a149ab637f52","Type":"ContainerStarted","Data":"4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce"}
Jul 24 00:19:30.660793 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:30.660773    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-djndv" event={"ID":"6699209e-9d46-4132-ae42-8bb20e8c0a51","Type":"ContainerStarted","Data":"fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b"}
Jul 24 00:19:30.673718 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:30.673665    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-x8clm" podStartSLOduration=1.27816495 podStartE2EDuration="4.673652119s" totalImagesPullingTime="3.395487169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:30.67233579 +0000 UTC m=+533.520654532" watchObservedRunningTime="2026-07-24 00:19:30.673652119 +0000 UTC m=+533.521970881"
Jul 24 00:19:30.684872 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:30.684815    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-djndv" podStartSLOduration=1.262423954 podStartE2EDuration="4.684799083s" totalImagesPullingTime="3.422375129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:30.684608581 +0000 UTC m=+533.532927323" watchObservedRunningTime="2026-07-24 00:19:30.684799083 +0000 UTC m=+533.533117824"
Jul 24 00:19:30.707874 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:30.707848    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-djndv"]
Jul 24 00:19:32.668144 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:32.668104    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-djndv" podUID="6699209e-9d46-4132-ae42-8bb20e8c0a51" containerName="authorino" containerID="cri-o://fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b" gracePeriod=30
Jul 24 00:19:32.911250 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:32.911226    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-djndv"
Jul 24 00:19:33.011385 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.011281    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9j9r\" (UniqueName: \"kubernetes.io/projected/6699209e-9d46-4132-ae42-8bb20e8c0a51-kube-api-access-s9j9r\") pod \"6699209e-9d46-4132-ae42-8bb20e8c0a51\" (UID: \"6699209e-9d46-4132-ae42-8bb20e8c0a51\") "
Jul 24 00:19:33.013400 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.013375    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6699209e-9d46-4132-ae42-8bb20e8c0a51-kube-api-access-s9j9r" (OuterVolumeSpecName: "kube-api-access-s9j9r") pod "6699209e-9d46-4132-ae42-8bb20e8c0a51" (UID: "6699209e-9d46-4132-ae42-8bb20e8c0a51"). InnerVolumeSpecName "kube-api-access-s9j9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:19:33.112669 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.112634    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9j9r\" (UniqueName: \"kubernetes.io/projected/6699209e-9d46-4132-ae42-8bb20e8c0a51-kube-api-access-s9j9r\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:19:33.672046 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.672010    2574 generic.go:356] "Generic (PLEG): container finished" podID="6699209e-9d46-4132-ae42-8bb20e8c0a51" containerID="fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b" exitCode=0
Jul 24 00:19:33.672472 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.672056    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-djndv"
Jul 24 00:19:33.672472 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.672096    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-djndv" event={"ID":"6699209e-9d46-4132-ae42-8bb20e8c0a51","Type":"ContainerDied","Data":"fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b"}
Jul 24 00:19:33.672472 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.672139    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-djndv" event={"ID":"6699209e-9d46-4132-ae42-8bb20e8c0a51","Type":"ContainerDied","Data":"3b3d01f75289f7dccf9d17705d88a30f3c3e7f539673c287073dcf0a0af9fe69"}
Jul 24 00:19:33.672472 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.672155    2574 scope.go:117] "RemoveContainer" containerID="fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b"
Jul 24 00:19:33.679807 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.679788    2574 scope.go:117] "RemoveContainer" containerID="fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b"
Jul 24 00:19:33.680078 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:19:33.680061    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b\": container with ID starting with fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b not found: ID does not exist" containerID="fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b"
Jul 24 00:19:33.680157 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.680086    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b"} err="failed to get container status \"fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b\": rpc error: code = NotFound desc = could not find container \"fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b\": container with ID starting with fc7750aa7772f4d63b5688a434c20641f946327b7b0532202eaa9b28bef2693b not found: ID does not exist"
Jul 24 00:19:33.691329 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.691293    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-djndv"]
Jul 24 00:19:33.694501 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.694482    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-djndv"]
Jul 24 00:19:33.750951 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:33.750921    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6699209e-9d46-4132-ae42-8bb20e8c0a51" path="/var/lib/kubelet/pods/6699209e-9d46-4132-ae42-8bb20e8c0a51/volumes"
Jul 24 00:19:55.278241 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.278203    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6bqvn"]
Jul 24 00:19:55.278758 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.278517    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6699209e-9d46-4132-ae42-8bb20e8c0a51" containerName="authorino"
Jul 24 00:19:55.278758 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.278528    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6699209e-9d46-4132-ae42-8bb20e8c0a51" containerName="authorino"
Jul 24 00:19:55.278758 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.278585    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6699209e-9d46-4132-ae42-8bb20e8c0a51" containerName="authorino"
Jul 24 00:19:55.284771 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.284750    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6bqvn"
Jul 24 00:19:55.287230 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.287205    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6bqvn"]
Jul 24 00:19:55.379177 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.379141    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hsk\" (UniqueName: \"kubernetes.io/projected/8bcd9663-c139-456e-a8cf-8a99d13950ae-kube-api-access-z4hsk\") pod \"authorino-5d4b67c446-6bqvn\" (UID: \"8bcd9663-c139-456e-a8cf-8a99d13950ae\") " pod="kuadrant-system/authorino-5d4b67c446-6bqvn"
Jul 24 00:19:55.480421 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.480382    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hsk\" (UniqueName: \"kubernetes.io/projected/8bcd9663-c139-456e-a8cf-8a99d13950ae-kube-api-access-z4hsk\") pod \"authorino-5d4b67c446-6bqvn\" (UID: \"8bcd9663-c139-456e-a8cf-8a99d13950ae\") " pod="kuadrant-system/authorino-5d4b67c446-6bqvn"
Jul 24 00:19:55.489616 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.489588    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hsk\" (UniqueName: \"kubernetes.io/projected/8bcd9663-c139-456e-a8cf-8a99d13950ae-kube-api-access-z4hsk\") pod \"authorino-5d4b67c446-6bqvn\" (UID: \"8bcd9663-c139-456e-a8cf-8a99d13950ae\") " pod="kuadrant-system/authorino-5d4b67c446-6bqvn"
Jul 24 00:19:55.546262 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.546188    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6bqvn"]
Jul 24 00:19:55.546431 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.546418    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6bqvn"
Jul 24 00:19:55.578490 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.578451    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c789db758-vgpk7"]
Jul 24 00:19:55.583131 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.583108    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c789db758-vgpk7"
Jul 24 00:19:55.585473 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.585453    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 24 00:19:55.588534 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.588508    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c789db758-vgpk7"]
Jul 24 00:19:55.681771 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.681714    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b5171b30-2c12-4025-8f55-aaaab923d531-tls-cert\") pod \"authorino-7c789db758-vgpk7\" (UID: \"b5171b30-2c12-4025-8f55-aaaab923d531\") " pod="kuadrant-system/authorino-7c789db758-vgpk7"
Jul 24 00:19:55.682208 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.681787    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjhnb\" (UniqueName: \"kubernetes.io/projected/b5171b30-2c12-4025-8f55-aaaab923d531-kube-api-access-cjhnb\") pod \"authorino-7c789db758-vgpk7\" (UID: \"b5171b30-2c12-4025-8f55-aaaab923d531\") " pod="kuadrant-system/authorino-7c789db758-vgpk7"
Jul 24 00:19:55.683714 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.683621    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6bqvn"]
Jul 24 00:19:55.686400 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:19:55.686367    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bcd9663_c139_456e_a8cf_8a99d13950ae.slice/crio-6df533cdd5b3275e09627e82e1784f6fd3a6b0c90f74689583a96d1f55ecfdef WatchSource:0}: Error finding container 6df533cdd5b3275e09627e82e1784f6fd3a6b0c90f74689583a96d1f55ecfdef: Status 404 returned error can't find the container with id 6df533cdd5b3275e09627e82e1784f6fd3a6b0c90f74689583a96d1f55ecfdef
Jul 24 00:19:55.752855 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.752825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6bqvn" event={"ID":"8bcd9663-c139-456e-a8cf-8a99d13950ae","Type":"ContainerStarted","Data":"6df533cdd5b3275e09627e82e1784f6fd3a6b0c90f74689583a96d1f55ecfdef"}
Jul 24 00:19:55.782659 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.782625    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b5171b30-2c12-4025-8f55-aaaab923d531-tls-cert\") pod \"authorino-7c789db758-vgpk7\" (UID: \"b5171b30-2c12-4025-8f55-aaaab923d531\") " pod="kuadrant-system/authorino-7c789db758-vgpk7"
Jul 24 00:19:55.782811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.782680    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhnb\" (UniqueName: \"kubernetes.io/projected/b5171b30-2c12-4025-8f55-aaaab923d531-kube-api-access-cjhnb\") pod \"authorino-7c789db758-vgpk7\" (UID: \"b5171b30-2c12-4025-8f55-aaaab923d531\") " pod="kuadrant-system/authorino-7c789db758-vgpk7"
Jul 24 00:19:55.784962 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.784943    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b5171b30-2c12-4025-8f55-aaaab923d531-tls-cert\") pod \"authorino-7c789db758-vgpk7\" (UID: \"b5171b30-2c12-4025-8f55-aaaab923d531\") " pod="kuadrant-system/authorino-7c789db758-vgpk7"
Jul 24 00:19:55.790140 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.790115    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhnb\" (UniqueName: \"kubernetes.io/projected/b5171b30-2c12-4025-8f55-aaaab923d531-kube-api-access-cjhnb\") pod \"authorino-7c789db758-vgpk7\" (UID: \"b5171b30-2c12-4025-8f55-aaaab923d531\") " pod="kuadrant-system/authorino-7c789db758-vgpk7"
Jul 24 00:19:55.896498 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:55.896468    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c789db758-vgpk7"
Jul 24 00:19:56.037063 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:56.037041    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c789db758-vgpk7"]
Jul 24 00:19:56.039376 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:19:56.039342    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5171b30_2c12_4025_8f55_aaaab923d531.slice/crio-ef1251775675d3bcf8706964e0f767f1e7a5b50f21f74e4e9ab84866490996d6 WatchSource:0}: Error finding container ef1251775675d3bcf8706964e0f767f1e7a5b50f21f74e4e9ab84866490996d6: Status 404 returned error can't find the container with id ef1251775675d3bcf8706964e0f767f1e7a5b50f21f74e4e9ab84866490996d6
Jul 24 00:19:56.758334 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:56.758215    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c789db758-vgpk7" event={"ID":"b5171b30-2c12-4025-8f55-aaaab923d531","Type":"ContainerStarted","Data":"f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00"}
Jul 24 00:19:56.758334 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:56.758256    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c789db758-vgpk7" event={"ID":"b5171b30-2c12-4025-8f55-aaaab923d531","Type":"ContainerStarted","Data":"ef1251775675d3bcf8706964e0f767f1e7a5b50f21f74e4e9ab84866490996d6"}
Jul 24 00:19:56.759661 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:56.759631    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6bqvn" event={"ID":"8bcd9663-c139-456e-a8cf-8a99d13950ae","Type":"ContainerStarted","Data":"65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf"}
Jul 24 00:19:56.759785 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:56.759674    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-6bqvn" podUID="8bcd9663-c139-456e-a8cf-8a99d13950ae" containerName="authorino" containerID="cri-o://65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf" gracePeriod=30
Jul 24 00:19:56.773023 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:56.772982    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c789db758-vgpk7" podStartSLOduration=1.414715736 podStartE2EDuration="1.772968229s" totalImagesPullingTime="358.252493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:56.772129433 +0000 UTC m=+559.620448184" watchObservedRunningTime="2026-07-24 00:19:56.772968229 +0000 UTC m=+559.621286970"
Jul 24 00:19:56.786268 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:56.786214    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-6bqvn" podStartSLOduration=1.306239468 podStartE2EDuration="1.786194003s" totalImagesPullingTime="479.954535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:56.785863205 +0000 UTC m=+559.634181947" watchObservedRunningTime="2026-07-24 00:19:56.786194003 +0000 UTC m=+559.634512745"
Jul 24 00:19:56.799953 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:56.799919    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x8clm"]
Jul 24 00:19:56.800144 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:56.800118    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-x8clm" podUID="e574c8d5-1312-4639-b4c1-a149ab637f52" containerName="authorino" containerID="cri-o://4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce" gracePeriod=30
Jul 24 00:19:57.048511 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.048489    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6bqvn"
Jul 24 00:19:57.051366 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.051347    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x8clm"
Jul 24 00:19:57.094549 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.094520    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4hsk\" (UniqueName: \"kubernetes.io/projected/8bcd9663-c139-456e-a8cf-8a99d13950ae-kube-api-access-z4hsk\") pod \"8bcd9663-c139-456e-a8cf-8a99d13950ae\" (UID: \"8bcd9663-c139-456e-a8cf-8a99d13950ae\") "
Jul 24 00:19:57.096473 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.096452    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcd9663-c139-456e-a8cf-8a99d13950ae-kube-api-access-z4hsk" (OuterVolumeSpecName: "kube-api-access-z4hsk") pod "8bcd9663-c139-456e-a8cf-8a99d13950ae" (UID: "8bcd9663-c139-456e-a8cf-8a99d13950ae"). InnerVolumeSpecName "kube-api-access-z4hsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:19:57.195070 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.195035    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8ssm\" (UniqueName: \"kubernetes.io/projected/e574c8d5-1312-4639-b4c1-a149ab637f52-kube-api-access-p8ssm\") pod \"e574c8d5-1312-4639-b4c1-a149ab637f52\" (UID: \"e574c8d5-1312-4639-b4c1-a149ab637f52\") "
Jul 24 00:19:57.195359 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.195341    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4hsk\" (UniqueName: \"kubernetes.io/projected/8bcd9663-c139-456e-a8cf-8a99d13950ae-kube-api-access-z4hsk\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:19:57.197042 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.197018    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e574c8d5-1312-4639-b4c1-a149ab637f52-kube-api-access-p8ssm" (OuterVolumeSpecName: "kube-api-access-p8ssm") pod "e574c8d5-1312-4639-b4c1-a149ab637f52" (UID: "e574c8d5-1312-4639-b4c1-a149ab637f52"). InnerVolumeSpecName "kube-api-access-p8ssm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:19:57.296495 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.296402    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ssm\" (UniqueName: \"kubernetes.io/projected/e574c8d5-1312-4639-b4c1-a149ab637f52-kube-api-access-p8ssm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:19:57.763736 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.763711    2574 generic.go:356] "Generic (PLEG): container finished" podID="8bcd9663-c139-456e-a8cf-8a99d13950ae" containerID="65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf" exitCode=0
Jul 24 00:19:57.764072 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.763750    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6bqvn"
Jul 24 00:19:57.764072 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.763793    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6bqvn" event={"ID":"8bcd9663-c139-456e-a8cf-8a99d13950ae","Type":"ContainerDied","Data":"65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf"}
Jul 24 00:19:57.764072 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.763827    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6bqvn" event={"ID":"8bcd9663-c139-456e-a8cf-8a99d13950ae","Type":"ContainerDied","Data":"6df533cdd5b3275e09627e82e1784f6fd3a6b0c90f74689583a96d1f55ecfdef"}
Jul 24 00:19:57.764072 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.763843    2574 scope.go:117] "RemoveContainer" containerID="65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf"
Jul 24 00:19:57.765034 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.765015    2574 generic.go:356] "Generic (PLEG): container finished" podID="e574c8d5-1312-4639-b4c1-a149ab637f52" containerID="4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce" exitCode=0
Jul 24 00:19:57.765131 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.765041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x8clm" event={"ID":"e574c8d5-1312-4639-b4c1-a149ab637f52","Type":"ContainerDied","Data":"4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce"}
Jul 24 00:19:57.765131 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.765066    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x8clm"
Jul 24 00:19:57.765131 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.765082    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x8clm" event={"ID":"e574c8d5-1312-4639-b4c1-a149ab637f52","Type":"ContainerDied","Data":"99a7ea75bb78b7184d0ab979e3d1acb8c277cc77c881cfdb4e90b403e8cb79cf"}
Jul 24 00:19:57.771811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.771796    2574 scope.go:117] "RemoveContainer" containerID="65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf"
Jul 24 00:19:57.772050 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:19:57.772031    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf\": container with ID starting with 65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf not found: ID does not exist" containerID="65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf"
Jul 24 00:19:57.772108 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.772058    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf"} err="failed to get container status \"65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf\": rpc error: code = NotFound desc = could not find container \"65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf\": container with ID starting with 65c0f5c073b02fb1a60af667232ad5eb9689697cf07a20093476ac777c73fddf not found: ID does not exist"
Jul 24 00:19:57.772108 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.772074    2574 scope.go:117] "RemoveContainer" containerID="4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce"
Jul 24 00:19:57.779039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.779021    2574 scope.go:117] "RemoveContainer" containerID="4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce"
Jul 24 00:19:57.779467 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:19:57.779444    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce\": container with ID starting with 4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce not found: ID does not exist" containerID="4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce"
Jul 24 00:19:57.779630 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.779476    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce"} err="failed to get container status \"4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce\": rpc error: code = NotFound desc = could not find container \"4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce\": container with ID starting with 4c7d340b7bc5600058d577a4efb67334d33231d24510af6d1d4891e46f7353ce not found: ID does not exist"
Jul 24 00:19:57.780690 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.780674    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6bqvn"]
Jul 24 00:19:57.784228 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.784210    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6bqvn"]
Jul 24 00:19:57.793489 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.793469    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x8clm"]
Jul 24 00:19:57.798785 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:57.798766    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x8clm"]
Jul 24 00:19:59.752093 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:59.752063    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bcd9663-c139-456e-a8cf-8a99d13950ae" path="/var/lib/kubelet/pods/8bcd9663-c139-456e-a8cf-8a99d13950ae/volumes"
Jul 24 00:19:59.752479 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:19:59.752393    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e574c8d5-1312-4639-b4c1-a149ab637f52" path="/var/lib/kubelet/pods/e574c8d5-1312-4639-b4c1-a149ab637f52/volumes"
Jul 24 00:20:39.385165 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.385135    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"]
Jul 24 00:20:39.385670 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.385390    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e574c8d5-1312-4639-b4c1-a149ab637f52" containerName="authorino"
Jul 24 00:20:39.385670 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.385400    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e574c8d5-1312-4639-b4c1-a149ab637f52" containerName="authorino"
Jul 24 00:20:39.385670 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.385408    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bcd9663-c139-456e-a8cf-8a99d13950ae" containerName="authorino"
Jul 24 00:20:39.385670 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.385413    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bcd9663-c139-456e-a8cf-8a99d13950ae" containerName="authorino"
Jul 24 00:20:39.385670 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.385474    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e574c8d5-1312-4639-b4c1-a149ab637f52" containerName="authorino"
Jul 24 00:20:39.385670 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.385481    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bcd9663-c139-456e-a8cf-8a99d13950ae" containerName="authorino"
Jul 24 00:20:39.389307 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.389289    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"
Jul 24 00:20:39.392644 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.392618    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-k7c2z\""
Jul 24 00:20:39.393706 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.393689    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 24 00:20:39.405591 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.405565    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"]
Jul 24 00:20:39.512982 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.512950    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb80b996-52a1-401c-beb3-9fe9b8ae3d54-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zs5fj\" (UID: \"eb80b996-52a1-401c-beb3-9fe9b8ae3d54\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"
Jul 24 00:20:39.513127 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.512996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhnh\" (UniqueName: \"kubernetes.io/projected/eb80b996-52a1-401c-beb3-9fe9b8ae3d54-kube-api-access-gbhnh\") pod \"payload-pre-processing-687dbb6cf6-zs5fj\" (UID: \"eb80b996-52a1-401c-beb3-9fe9b8ae3d54\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"
Jul 24 00:20:39.614043 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.614009    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhnh\" (UniqueName: \"kubernetes.io/projected/eb80b996-52a1-401c-beb3-9fe9b8ae3d54-kube-api-access-gbhnh\") pod \"payload-pre-processing-687dbb6cf6-zs5fj\" (UID: \"eb80b996-52a1-401c-beb3-9fe9b8ae3d54\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"
Jul 24 00:20:39.614224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.614069    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb80b996-52a1-401c-beb3-9fe9b8ae3d54-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zs5fj\" (UID: \"eb80b996-52a1-401c-beb3-9fe9b8ae3d54\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"
Jul 24 00:20:39.614703 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.614683    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb80b996-52a1-401c-beb3-9fe9b8ae3d54-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-zs5fj\" (UID: \"eb80b996-52a1-401c-beb3-9fe9b8ae3d54\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"
Jul 24 00:20:39.621537 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.621512    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhnh\" (UniqueName: \"kubernetes.io/projected/eb80b996-52a1-401c-beb3-9fe9b8ae3d54-kube-api-access-gbhnh\") pod \"payload-pre-processing-687dbb6cf6-zs5fj\" (UID: \"eb80b996-52a1-401c-beb3-9fe9b8ae3d54\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"
Jul 24 00:20:39.698956 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.698886    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"
Jul 24 00:20:39.822436 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.822361    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"]
Jul 24 00:20:39.824870 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:20:39.824844    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb80b996_52a1_401c_beb3_9fe9b8ae3d54.slice/crio-174aed558f5cb6d594dfe2458b20a98e5c7c0b43b45dd293871cf4c3879c1284 WatchSource:0}: Error finding container 174aed558f5cb6d594dfe2458b20a98e5c7c0b43b45dd293871cf4c3879c1284: Status 404 returned error can't find the container with id 174aed558f5cb6d594dfe2458b20a98e5c7c0b43b45dd293871cf4c3879c1284
Jul 24 00:20:39.915425 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:39.915394    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj" event={"ID":"eb80b996-52a1-401c-beb3-9fe9b8ae3d54","Type":"ContainerStarted","Data":"174aed558f5cb6d594dfe2458b20a98e5c7c0b43b45dd293871cf4c3879c1284"}
Jul 24 00:20:42.927935 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:42.927858    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj" event={"ID":"eb80b996-52a1-401c-beb3-9fe9b8ae3d54","Type":"ContainerStarted","Data":"dbc3b1614ee129307f40853fbae7498225bd83902a476a737d596e658033ec2c"}
Jul 24 00:20:42.928274 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:42.927987    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"
Jul 24 00:20:42.947530 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:42.947473    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj" podStartSLOduration=1.193943884 podStartE2EDuration="3.947455256s" totalImagesPullingTime="2.753511372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:20:42.943879462 +0000 UTC m=+605.792198202" watchObservedRunningTime="2026-07-24 00:20:42.947455256 +0000 UTC m=+605.795774004"
Jul 24 00:20:46.567440 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:46.567405    2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-mfq7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": read tcp 10.133.0.2:58784->10.133.0.19:15021: read: connection reset by peer" start-of-body=
Jul 24 00:20:46.567846 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:46.567472    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p" podUID="a88fbbbb-bb27-4049-99b0-31b75cb659ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": read tcp 10.133.0.2:58784->10.133.0.19:15021: read: connection reset by peer"
Jul 24 00:20:46.942848 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:46.942769    2574 generic.go:356] "Generic (PLEG): container finished" podID="a88fbbbb-bb27-4049-99b0-31b75cb659ed" containerID="72ebb2f1051aa5d23408b659e41b87bff5f77f6a69417eca5c28965486fdd87b" exitCode=0
Jul 24 00:20:46.942848 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:46.942837    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p" event={"ID":"a88fbbbb-bb27-4049-99b0-31b75cb659ed","Type":"ContainerDied","Data":"72ebb2f1051aa5d23408b659e41b87bff5f77f6a69417eca5c28965486fdd87b"}
Jul 24 00:20:46.943187 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:46.943174    2574 scope.go:117] "RemoveContainer" containerID="72ebb2f1051aa5d23408b659e41b87bff5f77f6a69417eca5c28965486fdd87b"
Jul 24 00:20:46.944084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:46.944060    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:20:46.944145 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:46.944127    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:20:46.944207 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:46.944159    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:20:47.950858 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:47.950821    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p" event={"ID":"a88fbbbb-bb27-4049-99b0-31b75cb659ed","Type":"ContainerStarted","Data":"1c2f650753d50c5f272c4e658f98506ccbe8bb77c1cff78e2b074c5f9b7b4910"}
Jul 24 00:20:48.081823 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:48.081786    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:20:48.082040 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:48.082015    2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-mfq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Jul 24 00:20:48.082086 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:48.082066    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p" podUID="a88fbbbb-bb27-4049-99b0-31b75cb659ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Jul 24 00:20:49.081572 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:49.081540    2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-mfq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Jul 24 00:20:49.081944 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:49.081592    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p" podUID="a88fbbbb-bb27-4049-99b0-31b75cb659ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Jul 24 00:20:50.081119 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:50.081080    2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-mfq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Jul 24 00:20:50.081348 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:50.081146    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p" podUID="a88fbbbb-bb27-4049-99b0-31b75cb659ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Jul 24 00:20:51.081856 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:51.081817    2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-mfq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Jul 24 00:20:51.082237 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:51.081873    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p" podUID="a88fbbbb-bb27-4049-99b0-31b75cb659ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Jul 24 00:20:52.084534 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:52.084501    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:20:52.084901 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:52.084737    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:20:52.085566 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:52.085549    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mfq7p"
Jul 24 00:20:53.933479 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:20:53.933451    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-zs5fj"
Jul 24 00:21:47.624872 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.624834    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"]
Jul 24 00:21:47.627870 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.627851    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.630558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.630537    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 00:21:47.631748 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.631729    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 24 00:21:47.631849 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.631813    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 00:21:47.631941 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.631928    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k8tlc\""
Jul 24 00:21:47.636610 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.636587    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"]
Jul 24 00:21:47.668521 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.668484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22463cb3-f2eb-4781-9715-8ff4595faf9f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.668521 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.668526    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.668752 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.668543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.668752 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.668600    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vsps\" (UniqueName: \"kubernetes.io/projected/22463cb3-f2eb-4781-9715-8ff4595faf9f-kube-api-access-5vsps\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.668752 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.668641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.668752 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.668668    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.769260 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.769227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22463cb3-f2eb-4781-9715-8ff4595faf9f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.769463 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.769266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.769463 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.769292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.769463 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.769347    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vsps\" (UniqueName: \"kubernetes.io/projected/22463cb3-f2eb-4781-9715-8ff4595faf9f-kube-api-access-5vsps\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.769463 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.769392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.769463 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.769428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.769819 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.769798    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.769886 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.769798    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.769886 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.769849    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.771676 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.771653    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22463cb3-f2eb-4781-9715-8ff4595faf9f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.771941 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.771919    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22463cb3-f2eb-4781-9715-8ff4595faf9f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.776882 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.776852    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vsps\" (UniqueName: \"kubernetes.io/projected/22463cb3-f2eb-4781-9715-8ff4595faf9f-kube-api-access-5vsps\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4df2v\" (UID: \"22463cb3-f2eb-4781-9715-8ff4595faf9f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:47.939541 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:47.939452    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:48.076536 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:48.076391    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"]
Jul 24 00:21:48.079202 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:21:48.079178    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22463cb3_f2eb_4781_9715_8ff4595faf9f.slice/crio-8138ba8d431eafa0395827dc3de90af687313c16a29785974e6f3b863bf9fa4e WatchSource:0}: Error finding container 8138ba8d431eafa0395827dc3de90af687313c16a29785974e6f3b863bf9fa4e: Status 404 returned error can't find the container with id 8138ba8d431eafa0395827dc3de90af687313c16a29785974e6f3b863bf9fa4e
Jul 24 00:21:48.080589 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:48.080568    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:21:48.166938 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:48.166896    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v" event={"ID":"22463cb3-f2eb-4781-9715-8ff4595faf9f","Type":"ContainerStarted","Data":"8138ba8d431eafa0395827dc3de90af687313c16a29785974e6f3b863bf9fa4e"}
Jul 24 00:21:49.171627 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:49.171593    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v" event={"ID":"22463cb3-f2eb-4781-9715-8ff4595faf9f","Type":"ContainerStarted","Data":"a1101e2d293c73b93ee12932a208aa4d8f6a9e81d34747982e8e942ad5683339"}
Jul 24 00:21:49.187747 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:49.187668    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v" podStartSLOduration=1.237212933 podStartE2EDuration="2.187652656s" totalImagesPullingTime="950.439723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:21:49.186372632 +0000 UTC m=+672.034691373" watchObservedRunningTime="2026-07-24 00:21:49.187652656 +0000 UTC m=+672.035971398"
Jul 24 00:21:54.432569 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.432507    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"]
Jul 24 00:21:54.435780 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.435759    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.438072 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.438048    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 24 00:21:54.445816 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.445788    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"]
Jul 24 00:21:54.521516 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.521479    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.521516 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.521521    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.521731 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.521553    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.521731 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.521572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9gxm\" (UniqueName: \"kubernetes.io/projected/c1ae7dfd-449c-4728-a134-d1271b09cdae-kube-api-access-d9gxm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.521731 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.521591    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.521731 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.521646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ae7dfd-449c-4728-a134-d1271b09cdae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.622026 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.621984    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ae7dfd-449c-4728-a134-d1271b09cdae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.622224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.622071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.622224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.622107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.622224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.622142    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.622224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.622172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9gxm\" (UniqueName: \"kubernetes.io/projected/c1ae7dfd-449c-4728-a134-d1271b09cdae-kube-api-access-d9gxm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.622224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.622201    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.622568 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.622545    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.622673 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.622589    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.622734 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.622669    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.624363 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.624343    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7dfd-449c-4728-a134-d1271b09cdae-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.624619 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.624602    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ae7dfd-449c-4728-a134-d1271b09cdae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.630103 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.630080    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9gxm\" (UniqueName: \"kubernetes.io/projected/c1ae7dfd-449c-4728-a134-d1271b09cdae-kube-api-access-d9gxm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p\" (UID: \"c1ae7dfd-449c-4728-a134-d1271b09cdae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.747093 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.746990    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:21:54.875710 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:54.875647    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"]
Jul 24 00:21:54.878265 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:21:54.878241    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1ae7dfd_449c_4728_a134_d1271b09cdae.slice/crio-76559b8808c4f0f00d80f5144d6b777fe56f706084fb1060531b60e313fa4651 WatchSource:0}: Error finding container 76559b8808c4f0f00d80f5144d6b777fe56f706084fb1060531b60e313fa4651: Status 404 returned error can't find the container with id 76559b8808c4f0f00d80f5144d6b777fe56f706084fb1060531b60e313fa4651
Jul 24 00:21:55.196943 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:55.196906    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p" event={"ID":"c1ae7dfd-449c-4728-a134-d1271b09cdae","Type":"ContainerStarted","Data":"8ce7e4c62559d763d2269c574e3bb7fc3672d88fd2a4d1d9a72da2a13a14391f"}
Jul 24 00:21:55.196943 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:55.196946    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p" event={"ID":"c1ae7dfd-449c-4728-a134-d1271b09cdae","Type":"ContainerStarted","Data":"76559b8808c4f0f00d80f5144d6b777fe56f706084fb1060531b60e313fa4651"}
Jul 24 00:21:55.214532 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:55.214480    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p" podStartSLOduration=1.035516887 podStartE2EDuration="1.214463977s" totalImagesPullingTime="178.94709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:21:55.213296674 +0000 UTC m=+678.061615415" watchObservedRunningTime="2026-07-24 00:21:55.214463977 +0000 UTC m=+678.062782717"
Jul 24 00:21:57.940233 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:57.940181    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:57.940709 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:57.940257    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:57.952794 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:57.952760    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:21:58.224257 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:21:58.224177    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4df2v"
Jul 24 00:22:04.747756 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:22:04.747720    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:22:04.747756 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:22:04.747764    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:22:04.760420 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:22:04.760386    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:22:05.245520 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:22:05.245490    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p"
Jul 24 00:23:20.942348 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:20.942303    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bcf8b5879-srpzq"]
Jul 24 00:23:20.944479 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:20.944461    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcf8b5879-srpzq"
Jul 24 00:23:20.951461 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:20.951441    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bcf8b5879-srpzq"]
Jul 24 00:23:20.998105 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:20.998080    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87t8r\" (UniqueName: \"kubernetes.io/projected/9e0f2154-96bd-4912-baca-59499712d479-kube-api-access-87t8r\") pod \"authorino-bcf8b5879-srpzq\" (UID: \"9e0f2154-96bd-4912-baca-59499712d479\") " pod="kuadrant-system/authorino-bcf8b5879-srpzq"
Jul 24 00:23:20.998216 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:20.998137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e0f2154-96bd-4912-baca-59499712d479-tls-cert\") pod \"authorino-bcf8b5879-srpzq\" (UID: \"9e0f2154-96bd-4912-baca-59499712d479\") " pod="kuadrant-system/authorino-bcf8b5879-srpzq"
Jul 24 00:23:21.098643 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:21.098618    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87t8r\" (UniqueName: \"kubernetes.io/projected/9e0f2154-96bd-4912-baca-59499712d479-kube-api-access-87t8r\") pod \"authorino-bcf8b5879-srpzq\" (UID: \"9e0f2154-96bd-4912-baca-59499712d479\") " pod="kuadrant-system/authorino-bcf8b5879-srpzq"
Jul 24 00:23:21.098740 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:21.098660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e0f2154-96bd-4912-baca-59499712d479-tls-cert\") pod \"authorino-bcf8b5879-srpzq\" (UID: \"9e0f2154-96bd-4912-baca-59499712d479\") " pod="kuadrant-system/authorino-bcf8b5879-srpzq"
Jul 24 00:23:21.101088 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:21.101057    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e0f2154-96bd-4912-baca-59499712d479-tls-cert\") pod \"authorino-bcf8b5879-srpzq\" (UID: \"9e0f2154-96bd-4912-baca-59499712d479\") " pod="kuadrant-system/authorino-bcf8b5879-srpzq"
Jul 24 00:23:21.106085 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:21.106067    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87t8r\" (UniqueName: \"kubernetes.io/projected/9e0f2154-96bd-4912-baca-59499712d479-kube-api-access-87t8r\") pod \"authorino-bcf8b5879-srpzq\" (UID: \"9e0f2154-96bd-4912-baca-59499712d479\") " pod="kuadrant-system/authorino-bcf8b5879-srpzq"
Jul 24 00:23:21.253583 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:21.253507    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bcf8b5879-srpzq"
Jul 24 00:23:21.375165 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:21.375130    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bcf8b5879-srpzq"]
Jul 24 00:23:21.378291 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:23:21.378267    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e0f2154_96bd_4912_baca_59499712d479.slice/crio-c348a0455c808dbad3d7f8990bb9bb003cdc1f3c79df33b675031f9c66b049b6 WatchSource:0}: Error finding container c348a0455c808dbad3d7f8990bb9bb003cdc1f3c79df33b675031f9c66b049b6: Status 404 returned error can't find the container with id c348a0455c808dbad3d7f8990bb9bb003cdc1f3c79df33b675031f9c66b049b6
Jul 24 00:23:21.531167 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:21.531079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcf8b5879-srpzq" event={"ID":"9e0f2154-96bd-4912-baca-59499712d479","Type":"ContainerStarted","Data":"c348a0455c808dbad3d7f8990bb9bb003cdc1f3c79df33b675031f9c66b049b6"}
Jul 24 00:23:22.538554 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:22.538514    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bcf8b5879-srpzq" event={"ID":"9e0f2154-96bd-4912-baca-59499712d479","Type":"ContainerStarted","Data":"8b24b88423923fc95f8f798c24de5f46eb5099b6a4a37f4a2acc98a01fdc3e11"}
Jul 24 00:23:22.554426 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:22.554352    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bcf8b5879-srpzq" podStartSLOduration=2.084395131 podStartE2EDuration="2.554333075s" totalImagesPullingTime="469.937944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:23:22.55393535 +0000 UTC m=+765.402254093" watchObservedRunningTime="2026-07-24 00:23:22.554333075 +0000 UTC m=+765.402651819"
Jul 24 00:23:22.580250 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:22.580214    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c789db758-vgpk7"]
Jul 24 00:23:22.580448 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:22.580413    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7c789db758-vgpk7" podUID="b5171b30-2c12-4025-8f55-aaaab923d531" containerName="authorino" containerID="cri-o://f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00" gracePeriod=30
Jul 24 00:23:22.836680 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:22.836652    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c789db758-vgpk7"
Jul 24 00:23:22.913333 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:22.913291    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b5171b30-2c12-4025-8f55-aaaab923d531-tls-cert\") pod \"b5171b30-2c12-4025-8f55-aaaab923d531\" (UID: \"b5171b30-2c12-4025-8f55-aaaab923d531\") "
Jul 24 00:23:22.913526 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:22.913396    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjhnb\" (UniqueName: \"kubernetes.io/projected/b5171b30-2c12-4025-8f55-aaaab923d531-kube-api-access-cjhnb\") pod \"b5171b30-2c12-4025-8f55-aaaab923d531\" (UID: \"b5171b30-2c12-4025-8f55-aaaab923d531\") "
Jul 24 00:23:22.915641 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:22.915609    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5171b30-2c12-4025-8f55-aaaab923d531-kube-api-access-cjhnb" (OuterVolumeSpecName: "kube-api-access-cjhnb") pod "b5171b30-2c12-4025-8f55-aaaab923d531" (UID: "b5171b30-2c12-4025-8f55-aaaab923d531"). InnerVolumeSpecName "kube-api-access-cjhnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:23:22.922609 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:22.922564    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5171b30-2c12-4025-8f55-aaaab923d531-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b5171b30-2c12-4025-8f55-aaaab923d531" (UID: "b5171b30-2c12-4025-8f55-aaaab923d531"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:23:23.014219 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.014179    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjhnb\" (UniqueName: \"kubernetes.io/projected/b5171b30-2c12-4025-8f55-aaaab923d531-kube-api-access-cjhnb\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:23:23.014219 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.014218    2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b5171b30-2c12-4025-8f55-aaaab923d531-tls-cert\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:23:23.543940 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.543848    2574 generic.go:356] "Generic (PLEG): container finished" podID="b5171b30-2c12-4025-8f55-aaaab923d531" containerID="f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00" exitCode=0
Jul 24 00:23:23.543940 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.543888    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c789db758-vgpk7"
Jul 24 00:23:23.543940 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.543925    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c789db758-vgpk7" event={"ID":"b5171b30-2c12-4025-8f55-aaaab923d531","Type":"ContainerDied","Data":"f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00"}
Jul 24 00:23:23.544505 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.543963    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c789db758-vgpk7" event={"ID":"b5171b30-2c12-4025-8f55-aaaab923d531","Type":"ContainerDied","Data":"ef1251775675d3bcf8706964e0f767f1e7a5b50f21f74e4e9ab84866490996d6"}
Jul 24 00:23:23.544505 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.543985    2574 scope.go:117] "RemoveContainer" containerID="f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00"
Jul 24 00:23:23.558969 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.558943    2574 scope.go:117] "RemoveContainer" containerID="f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00"
Jul 24 00:23:23.559256 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:23:23.559233    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00\": container with ID starting with f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00 not found: ID does not exist" containerID="f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00"
Jul 24 00:23:23.559425 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.559264    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00"} err="failed to get container status \"f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00\": rpc error: code = NotFound desc = could not find container \"f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00\": container with ID starting with f2530fb87604ce4b39e0b1e7a8d5c3eace133777f7973044d80876d72459ac00 not found: ID does not exist"
Jul 24 00:23:23.568977 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.568951    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c789db758-vgpk7"]
Jul 24 00:23:23.575541 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.575516    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c789db758-vgpk7"]
Jul 24 00:23:23.751525 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:23:23.751491    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5171b30-2c12-4025-8f55-aaaab923d531" path="/var/lib/kubelet/pods/b5171b30-2c12-4025-8f55-aaaab923d531/volumes"
Jul 24 00:33:52.941609 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.941577    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"]
Jul 24 00:33:52.942074 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.941866    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5171b30-2c12-4025-8f55-aaaab923d531" containerName="authorino"
Jul 24 00:33:52.942074 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.941878    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5171b30-2c12-4025-8f55-aaaab923d531" containerName="authorino"
Jul 24 00:33:52.942074 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.941951    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5171b30-2c12-4025-8f55-aaaab923d531" containerName="authorino"
Jul 24 00:33:52.944768 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.944751    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:52.947368 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.947347    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d56291cf-kserve-self-signed-certs\""
Jul 24 00:33:52.963540 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.963516    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"]
Jul 24 00:33:52.970566 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.970539    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-tmp-dir\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:52.970659 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.970590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-model-cache\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:52.970703 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.970653    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wd84\" (UniqueName: \"kubernetes.io/projected/456fab67-794f-4ee6-8b3e-163a786e9e2d-kube-api-access-4wd84\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:52.970737 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.970716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-dshm\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:52.970799 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.970780    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/456fab67-794f-4ee6-8b3e-163a786e9e2d-tls-certs\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:52.970849 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:52.970837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-home\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.071996 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.071964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-home\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.071996 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.071998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-tmp-dir\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.072230 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.072018    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-model-cache\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.072230 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.072049    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wd84\" (UniqueName: \"kubernetes.io/projected/456fab67-794f-4ee6-8b3e-163a786e9e2d-kube-api-access-4wd84\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.072230 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.072090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-dshm\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.072230 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.072128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/456fab67-794f-4ee6-8b3e-163a786e9e2d-tls-certs\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.072476 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.072406    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-home\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.072476 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.072455    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-tmp-dir\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.072551 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.072496    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-model-cache\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.074506 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.074473    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-dshm\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.074685 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.074670    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/456fab67-794f-4ee6-8b3e-163a786e9e2d-tls-certs\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.081245 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.081219    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wd84\" (UniqueName: \"kubernetes.io/projected/456fab67-794f-4ee6-8b3e-163a786e9e2d-kube-api-access-4wd84\") pod \"e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") " pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.254378 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.254261    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:33:53.380878 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:53.380849    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"]
Jul 24 00:33:53.384493 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:33:53.384464    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod456fab67_794f_4ee6_8b3e_163a786e9e2d.slice/crio-af2fe72ba9d96091f8ca18c053792c1019b5bf9dafa053305dba03c83da196c0 WatchSource:0}: Error finding container af2fe72ba9d96091f8ca18c053792c1019b5bf9dafa053305dba03c83da196c0: Status 404 returned error can't find the container with id af2fe72ba9d96091f8ca18c053792c1019b5bf9dafa053305dba03c83da196c0
Jul 24 00:33:54.020867 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:54.020826    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72" event={"ID":"456fab67-794f-4ee6-8b3e-163a786e9e2d","Type":"ContainerStarted","Data":"bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60"}
Jul 24 00:33:54.021365 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:54.020876    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72" event={"ID":"456fab67-794f-4ee6-8b3e-163a786e9e2d","Type":"ContainerStarted","Data":"af2fe72ba9d96091f8ca18c053792c1019b5bf9dafa053305dba03c83da196c0"}
Jul 24 00:33:54.038681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:33:54.038630    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72" podStartSLOduration=2.038614929 podStartE2EDuration="2.038614929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:33:54.037200236 +0000 UTC m=+1396.885518978" watchObservedRunningTime="2026-07-24 00:33:54.038614929 +0000 UTC m=+1396.886933669"
Jul 24 00:34:03.254607 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:03.254516    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:34:03.254607 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:03.254572    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:34:03.267257 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:03.267231    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:34:04.072991 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.072964    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:34:04.723092 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.723054    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"]
Jul 24 00:34:04.726594 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.726574    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.729097 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.729073    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d56291cf-kserve-self-signed-certs\""
Jul 24 00:34:04.735339 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.735296    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"]
Jul 24 00:34:04.875126 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.875088    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b4cf4d-148e-4654-b3fd-535ced242bf5-tls-certs\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.875304 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.875135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-dshm\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.875304 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.875154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhvh6\" (UniqueName: \"kubernetes.io/projected/21b4cf4d-148e-4654-b3fd-535ced242bf5-kube-api-access-jhvh6\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.875304 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.875210    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-home\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.875304 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.875263    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-model-cache\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.875474 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.875352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-tmp-dir\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.976713 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.976621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-tmp-dir\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.976713 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.976672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b4cf4d-148e-4654-b3fd-535ced242bf5-tls-certs\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.976929 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.976705    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-dshm\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.976929 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.976736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhvh6\" (UniqueName: \"kubernetes.io/projected/21b4cf4d-148e-4654-b3fd-535ced242bf5-kube-api-access-jhvh6\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.976929 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.976764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-home\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.976929 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.976793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-model-cache\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.977112 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.977046    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-tmp-dir\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.977179 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.977155    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-home\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.977219 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.977184    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-model-cache\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.979006 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.978981    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-dshm\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.979271 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.979253    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b4cf4d-148e-4654-b3fd-535ced242bf5-tls-certs\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:04.986232 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:04.986205    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhvh6\" (UniqueName: \"kubernetes.io/projected/21b4cf4d-148e-4654-b3fd-535ced242bf5-kube-api-access-jhvh6\") pod \"e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") " pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:05.037103 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:05.037063    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:05.171589 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:05.171557    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"]
Jul 24 00:34:05.174844 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:34:05.174815    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21b4cf4d_148e_4654_b3fd_535ced242bf5.slice/crio-1d6e33f455286fdc72b0a2c0251141e82cd81ff53ada7a70ceb2c1d3c144258d WatchSource:0}: Error finding container 1d6e33f455286fdc72b0a2c0251141e82cd81ff53ada7a70ceb2c1d3c144258d: Status 404 returned error can't find the container with id 1d6e33f455286fdc72b0a2c0251141e82cd81ff53ada7a70ceb2c1d3c144258d
Jul 24 00:34:06.070474 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:06.070437    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg" event={"ID":"21b4cf4d-148e-4654-b3fd-535ced242bf5","Type":"ContainerStarted","Data":"071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8"}
Jul 24 00:34:06.070474 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:06.070480    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg" event={"ID":"21b4cf4d-148e-4654-b3fd-535ced242bf5","Type":"ContainerStarted","Data":"1d6e33f455286fdc72b0a2c0251141e82cd81ff53ada7a70ceb2c1d3c144258d"}
Jul 24 00:34:06.088197 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:06.088148    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg" podStartSLOduration=2.088132789 podStartE2EDuration="2.088132789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:34:06.086834044 +0000 UTC m=+1408.935152791" watchObservedRunningTime="2026-07-24 00:34:06.088132789 +0000 UTC m=+1408.936451529"
Jul 24 00:34:15.037200 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:15.037162    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:15.037200 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:15.037212    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:15.049585 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:15.049555    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:15.118532 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:15.118499    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:39.485434 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:39.485354    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"]
Jul 24 00:34:41.224836 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:41.224800    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"]
Jul 24 00:34:54.504258 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.504220    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72" podUID="456fab67-794f-4ee6-8b3e-163a786e9e2d" containerName="main" containerID="cri-o://bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60" gracePeriod=45
Jul 24 00:34:54.748392 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.748366    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:34:54.811078 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.810999    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-tmp-dir\") pod \"456fab67-794f-4ee6-8b3e-163a786e9e2d\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") "
Jul 24 00:34:54.811078 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.811055    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-model-cache\") pod \"456fab67-794f-4ee6-8b3e-163a786e9e2d\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") "
Jul 24 00:34:54.811078 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.811074    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-dshm\") pod \"456fab67-794f-4ee6-8b3e-163a786e9e2d\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") "
Jul 24 00:34:54.811363 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.811095    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-home\") pod \"456fab67-794f-4ee6-8b3e-163a786e9e2d\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") "
Jul 24 00:34:54.811363 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.811158    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wd84\" (UniqueName: \"kubernetes.io/projected/456fab67-794f-4ee6-8b3e-163a786e9e2d-kube-api-access-4wd84\") pod \"456fab67-794f-4ee6-8b3e-163a786e9e2d\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") "
Jul 24 00:34:54.811363 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.811176    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/456fab67-794f-4ee6-8b3e-163a786e9e2d-tls-certs\") pod \"456fab67-794f-4ee6-8b3e-163a786e9e2d\" (UID: \"456fab67-794f-4ee6-8b3e-163a786e9e2d\") "
Jul 24 00:34:54.811363 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.811245    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "456fab67-794f-4ee6-8b3e-163a786e9e2d" (UID: "456fab67-794f-4ee6-8b3e-163a786e9e2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:34:54.811551 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.811389    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "456fab67-794f-4ee6-8b3e-163a786e9e2d" (UID: "456fab67-794f-4ee6-8b3e-163a786e9e2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:34:54.811551 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.811404    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-home" (OuterVolumeSpecName: "home") pod "456fab67-794f-4ee6-8b3e-163a786e9e2d" (UID: "456fab67-794f-4ee6-8b3e-163a786e9e2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:34:54.811551 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.811467    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:54.811551 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.811482    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:54.811551 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.811496    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:54.813458 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.813428    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456fab67-794f-4ee6-8b3e-163a786e9e2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "456fab67-794f-4ee6-8b3e-163a786e9e2d" (UID: "456fab67-794f-4ee6-8b3e-163a786e9e2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:34:54.813566 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.813472    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-dshm" (OuterVolumeSpecName: "dshm") pod "456fab67-794f-4ee6-8b3e-163a786e9e2d" (UID: "456fab67-794f-4ee6-8b3e-163a786e9e2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:34:54.813566 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.813486    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456fab67-794f-4ee6-8b3e-163a786e9e2d-kube-api-access-4wd84" (OuterVolumeSpecName: "kube-api-access-4wd84") pod "456fab67-794f-4ee6-8b3e-163a786e9e2d" (UID: "456fab67-794f-4ee6-8b3e-163a786e9e2d"). InnerVolumeSpecName "kube-api-access-4wd84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:34:54.912469 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.912438    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wd84\" (UniqueName: \"kubernetes.io/projected/456fab67-794f-4ee6-8b3e-163a786e9e2d-kube-api-access-4wd84\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:54.912469 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.912463    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/456fab67-794f-4ee6-8b3e-163a786e9e2d-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:54.912469 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:54.912474    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/456fab67-794f-4ee6-8b3e-163a786e9e2d-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:55.270398 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:55.270363    2574 generic.go:356] "Generic (PLEG): container finished" podID="456fab67-794f-4ee6-8b3e-163a786e9e2d" containerID="bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60" exitCode=0
Jul 24 00:34:55.270558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:55.270405    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"
Jul 24 00:34:55.270558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:55.270449    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72" event={"ID":"456fab67-794f-4ee6-8b3e-163a786e9e2d","Type":"ContainerDied","Data":"bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60"}
Jul 24 00:34:55.270558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:55.270492    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72" event={"ID":"456fab67-794f-4ee6-8b3e-163a786e9e2d","Type":"ContainerDied","Data":"af2fe72ba9d96091f8ca18c053792c1019b5bf9dafa053305dba03c83da196c0"}
Jul 24 00:34:55.270558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:55.270511    2574 scope.go:117] "RemoveContainer" containerID="bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60"
Jul 24 00:34:55.279376 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:55.279356    2574 scope.go:117] "RemoveContainer" containerID="bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60"
Jul 24 00:34:55.279630 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:34:55.279609    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60\": container with ID starting with bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60 not found: ID does not exist" containerID="bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60"
Jul 24 00:34:55.279681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:55.279640    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60"} err="failed to get container status \"bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60\": rpc error: code = NotFound desc = could not find container \"bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60\": container with ID starting with bbc031ee7f3816784de675b906c5cc10d73e6c6220281b7060432193c8b9ba60 not found: ID does not exist"
Jul 24 00:34:55.292226 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:55.292205    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"]
Jul 24 00:34:55.296100 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:55.296077    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d56291cf-kserve-86b49d9975-x8c72"]
Jul 24 00:34:55.753008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:55.752974    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="456fab67-794f-4ee6-8b3e-163a786e9e2d" path="/var/lib/kubelet/pods/456fab67-794f-4ee6-8b3e-163a786e9e2d/volumes"
Jul 24 00:34:56.243213 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.243174    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg" podUID="21b4cf4d-148e-4654-b3fd-535ced242bf5" containerName="main" containerID="cri-o://071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8" gracePeriod=45
Jul 24 00:34:56.490878 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.490853    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:56.628054 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.628016    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b4cf4d-148e-4654-b3fd-535ced242bf5-tls-certs\") pod \"21b4cf4d-148e-4654-b3fd-535ced242bf5\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") "
Jul 24 00:34:56.628054 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.628055    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-tmp-dir\") pod \"21b4cf4d-148e-4654-b3fd-535ced242bf5\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") "
Jul 24 00:34:56.628307 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.628078    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-dshm\") pod \"21b4cf4d-148e-4654-b3fd-535ced242bf5\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") "
Jul 24 00:34:56.628307 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.628128    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhvh6\" (UniqueName: \"kubernetes.io/projected/21b4cf4d-148e-4654-b3fd-535ced242bf5-kube-api-access-jhvh6\") pod \"21b4cf4d-148e-4654-b3fd-535ced242bf5\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") "
Jul 24 00:34:56.628307 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.628180    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-home\") pod \"21b4cf4d-148e-4654-b3fd-535ced242bf5\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") "
Jul 24 00:34:56.628307 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.628210    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-model-cache\") pod \"21b4cf4d-148e-4654-b3fd-535ced242bf5\" (UID: \"21b4cf4d-148e-4654-b3fd-535ced242bf5\") "
Jul 24 00:34:56.628532 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.628410    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21b4cf4d-148e-4654-b3fd-535ced242bf5" (UID: "21b4cf4d-148e-4654-b3fd-535ced242bf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:34:56.628532 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.628464    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-model-cache" (OuterVolumeSpecName: "model-cache") pod "21b4cf4d-148e-4654-b3fd-535ced242bf5" (UID: "21b4cf4d-148e-4654-b3fd-535ced242bf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:34:56.628532 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.628508    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:56.628681 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.628657    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-home" (OuterVolumeSpecName: "home") pod "21b4cf4d-148e-4654-b3fd-535ced242bf5" (UID: "21b4cf4d-148e-4654-b3fd-535ced242bf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:34:56.630368 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.630335    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b4cf4d-148e-4654-b3fd-535ced242bf5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21b4cf4d-148e-4654-b3fd-535ced242bf5" (UID: "21b4cf4d-148e-4654-b3fd-535ced242bf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:34:56.630368 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.630342    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-dshm" (OuterVolumeSpecName: "dshm") pod "21b4cf4d-148e-4654-b3fd-535ced242bf5" (UID: "21b4cf4d-148e-4654-b3fd-535ced242bf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:34:56.630518 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.630395    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b4cf4d-148e-4654-b3fd-535ced242bf5-kube-api-access-jhvh6" (OuterVolumeSpecName: "kube-api-access-jhvh6") pod "21b4cf4d-148e-4654-b3fd-535ced242bf5" (UID: "21b4cf4d-148e-4654-b3fd-535ced242bf5"). InnerVolumeSpecName "kube-api-access-jhvh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:34:56.729919 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.729885    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:56.729919 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.729914    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:56.729919 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.729925    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b4cf4d-148e-4654-b3fd-535ced242bf5-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:56.730143 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.729933    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b4cf4d-148e-4654-b3fd-535ced242bf5-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:56.730143 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:56.729942    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhvh6\" (UniqueName: \"kubernetes.io/projected/21b4cf4d-148e-4654-b3fd-535ced242bf5-kube-api-access-jhvh6\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:34:57.280472 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:57.280434    2574 generic.go:356] "Generic (PLEG): container finished" podID="21b4cf4d-148e-4654-b3fd-535ced242bf5" containerID="071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8" exitCode=0
Jul 24 00:34:57.280862 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:57.280487    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"
Jul 24 00:34:57.280862 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:57.280508    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg" event={"ID":"21b4cf4d-148e-4654-b3fd-535ced242bf5","Type":"ContainerDied","Data":"071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8"}
Jul 24 00:34:57.280862 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:57.280542    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg" event={"ID":"21b4cf4d-148e-4654-b3fd-535ced242bf5","Type":"ContainerDied","Data":"1d6e33f455286fdc72b0a2c0251141e82cd81ff53ada7a70ceb2c1d3c144258d"}
Jul 24 00:34:57.280862 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:57.280562    2574 scope.go:117] "RemoveContainer" containerID="071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8"
Jul 24 00:34:57.288889 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:57.288870    2574 scope.go:117] "RemoveContainer" containerID="071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8"
Jul 24 00:34:57.289134 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:34:57.289115    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8\": container with ID starting with 071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8 not found: ID does not exist" containerID="071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8"
Jul 24 00:34:57.289201 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:57.289141    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8"} err="failed to get container status \"071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8\": rpc error: code = NotFound desc = could not find container \"071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8\": container with ID starting with 071abf915ed896971edace41e38204bc67ae0f788e182858755692b68ef90ad8 not found: ID does not exist"
Jul 24 00:34:57.312751 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:57.312716    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"]
Jul 24 00:34:57.321115 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:57.321085    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d56291cf-kserve-655dc54887-d6xzg"]
Jul 24 00:34:57.752678 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:34:57.752642    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b4cf4d-148e-4654-b3fd-535ced242bf5" path="/var/lib/kubelet/pods/21b4cf4d-148e-4654-b3fd-535ced242bf5/volumes"
Jul 24 00:38:26.036588 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.036553    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"]
Jul 24 00:38:26.038946 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.036853    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456fab67-794f-4ee6-8b3e-163a786e9e2d" containerName="main"
Jul 24 00:38:26.038946 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.036864    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="456fab67-794f-4ee6-8b3e-163a786e9e2d" containerName="main"
Jul 24 00:38:26.038946 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.036884    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b4cf4d-148e-4654-b3fd-535ced242bf5" containerName="main"
Jul 24 00:38:26.038946 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.036889    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b4cf4d-148e-4654-b3fd-535ced242bf5" containerName="main"
Jul 24 00:38:26.038946 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.036934    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="456fab67-794f-4ee6-8b3e-163a786e9e2d" containerName="main"
Jul 24 00:38:26.038946 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.036941    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b4cf4d-148e-4654-b3fd-535ced242bf5" containerName="main"
Jul 24 00:38:26.039759 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.039740    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:26.043154 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.043134    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-42e5f403-serving-cert\""
Jul 24 00:38:26.043258 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.043239    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 00:38:26.043596 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.043579    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 00:38:26.044278 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.044264    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nv7dj\""
Jul 24 00:38:26.048715 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.048692    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"]
Jul 24 00:38:26.182184 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.182148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bfw\" (UniqueName: \"kubernetes.io/projected/f4b817da-3d22-4e7a-add5-e68b61020f14-kube-api-access-56bfw\") pod \"maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:26.182395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.182216    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b817da-3d22-4e7a-add5-e68b61020f14-maas-api-tls\") pod \"maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:26.182395 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.182237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b817da-3d22-4e7a-add5-e68b61020f14-openshift-service-ca\") pod \"maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:26.283618 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.283583    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b817da-3d22-4e7a-add5-e68b61020f14-maas-api-tls\") pod \"maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:26.283618 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.283618    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b817da-3d22-4e7a-add5-e68b61020f14-openshift-service-ca\") pod \"maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:26.283821 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.283751    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bfw\" (UniqueName: \"kubernetes.io/projected/f4b817da-3d22-4e7a-add5-e68b61020f14-kube-api-access-56bfw\") pod \"maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:26.284202 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.284171    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b817da-3d22-4e7a-add5-e68b61020f14-openshift-service-ca\") pod \"maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:26.286077 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.286058    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b817da-3d22-4e7a-add5-e68b61020f14-maas-api-tls\") pod \"maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:26.292620 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.292567    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bfw\" (UniqueName: \"kubernetes.io/projected/f4b817da-3d22-4e7a-add5-e68b61020f14-kube-api-access-56bfw\") pod \"maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:26.350570 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.350540    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:26.481217 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.481183    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"]
Jul 24 00:38:26.484007 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:38:26.483978    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b817da_3d22_4e7a_add5_e68b61020f14.slice/crio-2bc69df4ea4b32310120dcf3c32b58a09f4ca031887bb5e24a01e647adead3eb WatchSource:0}: Error finding container 2bc69df4ea4b32310120dcf3c32b58a09f4ca031887bb5e24a01e647adead3eb: Status 404 returned error can't find the container with id 2bc69df4ea4b32310120dcf3c32b58a09f4ca031887bb5e24a01e647adead3eb
Jul 24 00:38:26.485539 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.485519    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:38:26.649925 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.649890    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"]
Jul 24 00:38:26.654217 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.654201    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:38:26.657953 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.657935    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-42e5f403\""
Jul 24 00:38:26.669243 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.669222    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"]
Jul 24 00:38:26.692075 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.692042    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"]
Jul 24 00:38:26.695122 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.695103    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:38:26.698022 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.698005    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-42e5f403-dockercfg-59pqz\""
Jul 24 00:38:26.708849 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.708823    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"]
Jul 24 00:38:26.787559 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.787518    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65290f4f-30b8-4554-a038-aa0cd1b6db88-plugins-config-volume\") pod \"payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h\" (UID: \"65290f4f-30b8-4554-a038-aa0cd1b6db88\") " pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:38:26.787739 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.787567    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b165be-f410-4f9f-b3ff-38914443d3df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z\" (UID: \"e6b165be-f410-4f9f-b3ff-38914443d3df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:38:26.787739 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.787612    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lsw4\" (UniqueName: \"kubernetes.io/projected/65290f4f-30b8-4554-a038-aa0cd1b6db88-kube-api-access-9lsw4\") pod \"payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h\" (UID: \"65290f4f-30b8-4554-a038-aa0cd1b6db88\") " pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:38:26.787739 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.787662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvftb\" (UniqueName: \"kubernetes.io/projected/e6b165be-f410-4f9f-b3ff-38914443d3df-kube-api-access-cvftb\") pod \"payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z\" (UID: \"e6b165be-f410-4f9f-b3ff-38914443d3df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:38:26.888410 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.888376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvftb\" (UniqueName: \"kubernetes.io/projected/e6b165be-f410-4f9f-b3ff-38914443d3df-kube-api-access-cvftb\") pod \"payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z\" (UID: \"e6b165be-f410-4f9f-b3ff-38914443d3df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:38:26.888618 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.888440    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65290f4f-30b8-4554-a038-aa0cd1b6db88-plugins-config-volume\") pod \"payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h\" (UID: \"65290f4f-30b8-4554-a038-aa0cd1b6db88\") " pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:38:26.888618 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.888473    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b165be-f410-4f9f-b3ff-38914443d3df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z\" (UID: \"e6b165be-f410-4f9f-b3ff-38914443d3df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:38:26.888618 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.888491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lsw4\" (UniqueName: \"kubernetes.io/projected/65290f4f-30b8-4554-a038-aa0cd1b6db88-kube-api-access-9lsw4\") pod \"payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h\" (UID: \"65290f4f-30b8-4554-a038-aa0cd1b6db88\") " pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:38:26.889353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.889301    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65290f4f-30b8-4554-a038-aa0cd1b6db88-plugins-config-volume\") pod \"payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h\" (UID: \"65290f4f-30b8-4554-a038-aa0cd1b6db88\") " pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:38:26.889353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.889347    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b165be-f410-4f9f-b3ff-38914443d3df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z\" (UID: \"e6b165be-f410-4f9f-b3ff-38914443d3df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:38:26.899202 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.899180    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvftb\" (UniqueName: \"kubernetes.io/projected/e6b165be-f410-4f9f-b3ff-38914443d3df-kube-api-access-cvftb\") pod \"payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z\" (UID: \"e6b165be-f410-4f9f-b3ff-38914443d3df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:38:26.899301 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.899256    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lsw4\" (UniqueName: \"kubernetes.io/projected/65290f4f-30b8-4554-a038-aa0cd1b6db88-kube-api-access-9lsw4\") pod \"payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h\" (UID: \"65290f4f-30b8-4554-a038-aa0cd1b6db88\") " pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:38:26.963894 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:26.963821    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:38:27.004013 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:27.003982    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:38:27.102949 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:27.102916    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"]
Jul 24 00:38:27.107559 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:38:27.107524    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b165be_f410_4f9f_b3ff_38914443d3df.slice/crio-3f104593085bdd045efb44194423c598ed89b98126e75a6148852d2dc46286a9 WatchSource:0}: Error finding container 3f104593085bdd045efb44194423c598ed89b98126e75a6148852d2dc46286a9: Status 404 returned error can't find the container with id 3f104593085bdd045efb44194423c598ed89b98126e75a6148852d2dc46286a9
Jul 24 00:38:27.117330 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:27.117284    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w" event={"ID":"f4b817da-3d22-4e7a-add5-e68b61020f14","Type":"ContainerStarted","Data":"2bc69df4ea4b32310120dcf3c32b58a09f4ca031887bb5e24a01e647adead3eb"}
Jul 24 00:38:27.118699 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:27.118670    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z" event={"ID":"e6b165be-f410-4f9f-b3ff-38914443d3df","Type":"ContainerStarted","Data":"3f104593085bdd045efb44194423c598ed89b98126e75a6148852d2dc46286a9"}
Jul 24 00:38:27.145307 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:27.145279    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"]
Jul 24 00:38:27.148295 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:38:27.148267    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65290f4f_30b8_4554_a038_aa0cd1b6db88.slice/crio-a7a47bdee0a95bc669643c84879e20b23ce995e18e9f6796a2380d40d25f0517 WatchSource:0}: Error finding container a7a47bdee0a95bc669643c84879e20b23ce995e18e9f6796a2380d40d25f0517: Status 404 returned error can't find the container with id a7a47bdee0a95bc669643c84879e20b23ce995e18e9f6796a2380d40d25f0517
Jul 24 00:38:28.124642 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:28.124603    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h" event={"ID":"65290f4f-30b8-4554-a038-aa0cd1b6db88","Type":"ContainerStarted","Data":"02d2741ac16a7dd61bc1587b94be1e77552a868df53f84ec5b307d742e8b3a05"}
Jul 24 00:38:28.125118 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:28.124676    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h" event={"ID":"65290f4f-30b8-4554-a038-aa0cd1b6db88","Type":"ContainerStarted","Data":"a7a47bdee0a95bc669643c84879e20b23ce995e18e9f6796a2380d40d25f0517"}
Jul 24 00:38:28.125118 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:28.124711    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:38:28.126375 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:28.126343    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z" event={"ID":"e6b165be-f410-4f9f-b3ff-38914443d3df","Type":"ContainerStarted","Data":"38b84f6ab35c49d12d8d10d3423b07885399ee77093f7dcf93ce47f78b06d2fd"}
Jul 24 00:38:28.126528 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:28.126404    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:38:28.142195 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:28.142146    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h" podStartSLOduration=2.142129623 podStartE2EDuration="2.142129623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:38:28.14015299 +0000 UTC m=+1670.988471732" watchObservedRunningTime="2026-07-24 00:38:28.142129623 +0000 UTC m=+1670.990448364"
Jul 24 00:38:28.155155 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:28.155102    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z" podStartSLOduration=2.155081757 podStartE2EDuration="2.155081757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:38:28.154684048 +0000 UTC m=+1671.003002792" watchObservedRunningTime="2026-07-24 00:38:28.155081757 +0000 UTC m=+1671.003400499"
Jul 24 00:38:29.132305 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:29.132270    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w" event={"ID":"f4b817da-3d22-4e7a-add5-e68b61020f14","Type":"ContainerStarted","Data":"7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5"}
Jul 24 00:38:29.132715 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:29.132494    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:29.148547 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:29.148494    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w" podStartSLOduration=0.701529267 podStartE2EDuration="3.148473927s" totalImagesPullingTime="2.44694466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:38:29.148014099 +0000 UTC m=+1671.996332841" watchObservedRunningTime="2026-07-24 00:38:29.148473927 +0000 UTC m=+1671.996792672"
Jul 24 00:38:35.142142 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:35.142112    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:38:39.134501 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:39.134467    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:38:39.134915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:38:39.134540    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:39:01.993257 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:01.993171    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"]
Jul 24 00:39:01.993788 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:01.993450    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w" podUID="f4b817da-3d22-4e7a-add5-e68b61020f14" containerName="maas-api" containerID="cri-o://7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5" gracePeriod=30
Jul 24 00:39:02.097992 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.097960    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"]
Jul 24 00:39:02.098273 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.098216    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h" podUID="65290f4f-30b8-4554-a038-aa0cd1b6db88" containerName="payload-processing" containerID="cri-o://02d2741ac16a7dd61bc1587b94be1e77552a868df53f84ec5b307d742e8b3a05" gracePeriod=30
Jul 24 00:39:02.127744 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.127670    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"]
Jul 24 00:39:02.128885 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.127900    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z" podUID="e6b165be-f410-4f9f-b3ff-38914443d3df" containerName="payload-pre-processing" containerID="cri-o://38b84f6ab35c49d12d8d10d3423b07885399ee77093f7dcf93ce47f78b06d2fd" gracePeriod=30
Jul 24 00:39:02.248143 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.248064    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:39:02.262940 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.262903    2574 generic.go:356] "Generic (PLEG): container finished" podID="f4b817da-3d22-4e7a-add5-e68b61020f14" containerID="7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5" exitCode=0
Jul 24 00:39:02.263127 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.262954    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"
Jul 24 00:39:02.263127 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.262985    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w" event={"ID":"f4b817da-3d22-4e7a-add5-e68b61020f14","Type":"ContainerDied","Data":"7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5"}
Jul 24 00:39:02.263127 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.263049    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w" event={"ID":"f4b817da-3d22-4e7a-add5-e68b61020f14","Type":"ContainerDied","Data":"2bc69df4ea4b32310120dcf3c32b58a09f4ca031887bb5e24a01e647adead3eb"}
Jul 24 00:39:02.263127 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.263084    2574 scope.go:117] "RemoveContainer" containerID="7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5"
Jul 24 00:39:02.265030 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.265001    2574 generic.go:356] "Generic (PLEG): container finished" podID="65290f4f-30b8-4554-a038-aa0cd1b6db88" containerID="02d2741ac16a7dd61bc1587b94be1e77552a868df53f84ec5b307d742e8b3a05" exitCode=0
Jul 24 00:39:02.265143 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.265088    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h" event={"ID":"65290f4f-30b8-4554-a038-aa0cd1b6db88","Type":"ContainerDied","Data":"02d2741ac16a7dd61bc1587b94be1e77552a868df53f84ec5b307d742e8b3a05"}
Jul 24 00:39:02.268601 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.268573    2574 generic.go:356] "Generic (PLEG): container finished" podID="e6b165be-f410-4f9f-b3ff-38914443d3df" containerID="38b84f6ab35c49d12d8d10d3423b07885399ee77093f7dcf93ce47f78b06d2fd" exitCode=0
Jul 24 00:39:02.268723 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.268676    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z" event={"ID":"e6b165be-f410-4f9f-b3ff-38914443d3df","Type":"ContainerDied","Data":"38b84f6ab35c49d12d8d10d3423b07885399ee77093f7dcf93ce47f78b06d2fd"}
Jul 24 00:39:02.276440 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.276421    2574 scope.go:117] "RemoveContainer" containerID="7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5"
Jul 24 00:39:02.276901 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:39:02.276875    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5\": container with ID starting with 7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5 not found: ID does not exist" containerID="7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5"
Jul 24 00:39:02.276990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.276911    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5"} err="failed to get container status \"7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5\": rpc error: code = NotFound desc = could not find container \"7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5\": container with ID starting with 7ae731ff95321667dad3b02028c8b0a45f5f35b1dc07245209fe1d10e9116ca5 not found: ID does not exist"
Jul 24 00:39:02.283149 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.283132    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b817da-3d22-4e7a-add5-e68b61020f14-openshift-service-ca\") pod \"f4b817da-3d22-4e7a-add5-e68b61020f14\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") "
Jul 24 00:39:02.283594 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.283489    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b817da-3d22-4e7a-add5-e68b61020f14-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f4b817da-3d22-4e7a-add5-e68b61020f14" (UID: "f4b817da-3d22-4e7a-add5-e68b61020f14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:39:02.283904 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.283559    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56bfw\" (UniqueName: \"kubernetes.io/projected/f4b817da-3d22-4e7a-add5-e68b61020f14-kube-api-access-56bfw\") pod \"f4b817da-3d22-4e7a-add5-e68b61020f14\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") "
Jul 24 00:39:02.284060 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.284045    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b817da-3d22-4e7a-add5-e68b61020f14-maas-api-tls\") pod \"f4b817da-3d22-4e7a-add5-e68b61020f14\" (UID: \"f4b817da-3d22-4e7a-add5-e68b61020f14\") "
Jul 24 00:39:02.284862 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.284847    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b817da-3d22-4e7a-add5-e68b61020f14-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:02.285977 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.285952    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b817da-3d22-4e7a-add5-e68b61020f14-kube-api-access-56bfw" (OuterVolumeSpecName: "kube-api-access-56bfw") pod "f4b817da-3d22-4e7a-add5-e68b61020f14" (UID: "f4b817da-3d22-4e7a-add5-e68b61020f14"). InnerVolumeSpecName "kube-api-access-56bfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:39:02.286853 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.286826    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b817da-3d22-4e7a-add5-e68b61020f14-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f4b817da-3d22-4e7a-add5-e68b61020f14" (UID: "f4b817da-3d22-4e7a-add5-e68b61020f14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:39:02.385412 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.385392    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56bfw\" (UniqueName: \"kubernetes.io/projected/f4b817da-3d22-4e7a-add5-e68b61020f14-kube-api-access-56bfw\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:02.385412 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.385413    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b817da-3d22-4e7a-add5-e68b61020f14-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:02.389839 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.389818    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:39:02.396336 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.391924    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:39:02.486209 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.486174    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b165be-f410-4f9f-b3ff-38914443d3df-plugins-config-volume\") pod \"e6b165be-f410-4f9f-b3ff-38914443d3df\" (UID: \"e6b165be-f410-4f9f-b3ff-38914443d3df\") "
Jul 24 00:39:02.486397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.486261    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lsw4\" (UniqueName: \"kubernetes.io/projected/65290f4f-30b8-4554-a038-aa0cd1b6db88-kube-api-access-9lsw4\") pod \"65290f4f-30b8-4554-a038-aa0cd1b6db88\" (UID: \"65290f4f-30b8-4554-a038-aa0cd1b6db88\") "
Jul 24 00:39:02.486397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.486283    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65290f4f-30b8-4554-a038-aa0cd1b6db88-plugins-config-volume\") pod \"65290f4f-30b8-4554-a038-aa0cd1b6db88\" (UID: \"65290f4f-30b8-4554-a038-aa0cd1b6db88\") "
Jul 24 00:39:02.486397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.486331    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvftb\" (UniqueName: \"kubernetes.io/projected/e6b165be-f410-4f9f-b3ff-38914443d3df-kube-api-access-cvftb\") pod \"e6b165be-f410-4f9f-b3ff-38914443d3df\" (UID: \"e6b165be-f410-4f9f-b3ff-38914443d3df\") "
Jul 24 00:39:02.486640 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.486616    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65290f4f-30b8-4554-a038-aa0cd1b6db88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65290f4f-30b8-4554-a038-aa0cd1b6db88" (UID: "65290f4f-30b8-4554-a038-aa0cd1b6db88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:39:02.486684 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.486636    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b165be-f410-4f9f-b3ff-38914443d3df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6b165be-f410-4f9f-b3ff-38914443d3df" (UID: "e6b165be-f410-4f9f-b3ff-38914443d3df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:39:02.488518 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.488494    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65290f4f-30b8-4554-a038-aa0cd1b6db88-kube-api-access-9lsw4" (OuterVolumeSpecName: "kube-api-access-9lsw4") pod "65290f4f-30b8-4554-a038-aa0cd1b6db88" (UID: "65290f4f-30b8-4554-a038-aa0cd1b6db88"). InnerVolumeSpecName "kube-api-access-9lsw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:39:02.488587 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.488570    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b165be-f410-4f9f-b3ff-38914443d3df-kube-api-access-cvftb" (OuterVolumeSpecName: "kube-api-access-cvftb") pod "e6b165be-f410-4f9f-b3ff-38914443d3df" (UID: "e6b165be-f410-4f9f-b3ff-38914443d3df"). InnerVolumeSpecName "kube-api-access-cvftb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:39:02.584142 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.584114    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"]
Jul 24 00:39:02.586981 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.586949    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lsw4\" (UniqueName: \"kubernetes.io/projected/65290f4f-30b8-4554-a038-aa0cd1b6db88-kube-api-access-9lsw4\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:02.586981 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.586978    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65290f4f-30b8-4554-a038-aa0cd1b6db88-plugins-config-volume\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:02.587171 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.586992    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvftb\" (UniqueName: \"kubernetes.io/projected/e6b165be-f410-4f9f-b3ff-38914443d3df-kube-api-access-cvftb\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:02.587171 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.587012    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b165be-f410-4f9f-b3ff-38914443d3df-plugins-config-volume\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:02.588217 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:02.588197    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-42e5f403-d6dfcd7b8-2q24w"]
Jul 24 00:39:03.274212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.274120    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"
Jul 24 00:39:03.274212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.274140    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z" event={"ID":"e6b165be-f410-4f9f-b3ff-38914443d3df","Type":"ContainerDied","Data":"3f104593085bdd045efb44194423c598ed89b98126e75a6148852d2dc46286a9"}
Jul 24 00:39:03.274212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.274208    2574 scope.go:117] "RemoveContainer" containerID="38b84f6ab35c49d12d8d10d3423b07885399ee77093f7dcf93ce47f78b06d2fd"
Jul 24 00:39:03.276459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.276431    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h" event={"ID":"65290f4f-30b8-4554-a038-aa0cd1b6db88","Type":"ContainerDied","Data":"a7a47bdee0a95bc669643c84879e20b23ce995e18e9f6796a2380d40d25f0517"}
Jul 24 00:39:03.276557 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.276479    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"
Jul 24 00:39:03.286393 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.286376    2574 scope.go:117] "RemoveContainer" containerID="02d2741ac16a7dd61bc1587b94be1e77552a868df53f84ec5b307d742e8b3a05"
Jul 24 00:39:03.297192 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.297166    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"]
Jul 24 00:39:03.300069 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.300049    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-42e5f403-5c8cb8db4d-n5n8z"]
Jul 24 00:39:03.310011 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.309988    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"]
Jul 24 00:39:03.313767 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.313747    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-42e5f403-67599bdf98-8b56h"]
Jul 24 00:39:03.752892 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.752855    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65290f4f-30b8-4554-a038-aa0cd1b6db88" path="/var/lib/kubelet/pods/65290f4f-30b8-4554-a038-aa0cd1b6db88/volumes"
Jul 24 00:39:03.753211 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.753198    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6b165be-f410-4f9f-b3ff-38914443d3df" path="/var/lib/kubelet/pods/e6b165be-f410-4f9f-b3ff-38914443d3df/volumes"
Jul 24 00:39:03.753558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:03.753545    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4b817da-3d22-4e7a-add5-e68b61020f14" path="/var/lib/kubelet/pods/f4b817da-3d22-4e7a-add5-e68b61020f14/volumes"
Jul 24 00:39:07.890679 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.890644    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"]
Jul 24 00:39:07.891125 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.890916    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b165be-f410-4f9f-b3ff-38914443d3df" containerName="payload-pre-processing"
Jul 24 00:39:07.891125 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.890926    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b165be-f410-4f9f-b3ff-38914443d3df" containerName="payload-pre-processing"
Jul 24 00:39:07.891125 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.890936    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65290f4f-30b8-4554-a038-aa0cd1b6db88" containerName="payload-processing"
Jul 24 00:39:07.891125 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.890941    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="65290f4f-30b8-4554-a038-aa0cd1b6db88" containerName="payload-processing"
Jul 24 00:39:07.891125 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.890953    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b817da-3d22-4e7a-add5-e68b61020f14" containerName="maas-api"
Jul 24 00:39:07.891125 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.890958    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b817da-3d22-4e7a-add5-e68b61020f14" containerName="maas-api"
Jul 24 00:39:07.891125 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.891014    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="65290f4f-30b8-4554-a038-aa0cd1b6db88" containerName="payload-processing"
Jul 24 00:39:07.891125 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.891022    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b817da-3d22-4e7a-add5-e68b61020f14" containerName="maas-api"
Jul 24 00:39:07.891125 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.891029    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6b165be-f410-4f9f-b3ff-38914443d3df" containerName="payload-pre-processing"
Jul 24 00:39:07.896639 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.896621    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:07.900999 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.900971    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8405249f-gw-openshift-default-dockercfg-d4mx8\""
Jul 24 00:39:07.911768 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.911740    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"]
Jul 24 00:39:07.929240 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.929210    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-credential-socket\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:07.929364 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.929264    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-socket\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:07.929364 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.929298    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-envoy\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:07.929446 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.929366    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-token\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:07.929446 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.929401    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e4b380-d10c-4d5d-b399-7797eb58b804-istiod-ca-cert\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:07.929527 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.929449    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-data\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:07.929527 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.929473    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkqfx\" (UniqueName: \"kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-kube-api-access-qkqfx\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:07.929600 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.929522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-certs\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:07.929600 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:07.929572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-podinfo\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030251 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030220    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-credential-socket\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030268    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-socket\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-envoy\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030345    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-token\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030377    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e4b380-d10c-4d5d-b399-7797eb58b804-istiod-ca-cert\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030419    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-data\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkqfx\" (UniqueName: \"kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-kube-api-access-qkqfx\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030759 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-certs\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030759 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030514    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-podinfo\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030759 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030618    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-credential-socket\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030759 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030680    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-socket\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.030970 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.030947    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-certs\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.031156 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.031129    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-data\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.031356 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.031335    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e4b380-d10c-4d5d-b399-7797eb58b804-istiod-ca-cert\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.032760 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.032736    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-envoy\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.033078 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.033056    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-podinfo\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.039607 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.039579    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-token\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.041565 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.041539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkqfx\" (UniqueName: \"kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-kube-api-access-qkqfx\") pod \"e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") " pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.206773 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.206680    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:08.338000 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.337973    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"]
Jul 24 00:39:08.340113 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:39:08.340078    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e4b380_d10c_4d5d_b399_7797eb58b804.slice/crio-f4711f8f60ef59138bbae2d063045c8a0c2d4b3b35446d0548b51bdc15f2b627 WatchSource:0}: Error finding container f4711f8f60ef59138bbae2d063045c8a0c2d4b3b35446d0548b51bdc15f2b627: Status 404 returned error can't find the container with id f4711f8f60ef59138bbae2d063045c8a0c2d4b3b35446d0548b51bdc15f2b627
Jul 24 00:39:08.342171 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.342143    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:39:08.342259 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.342202    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:39:08.342259 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:08.342229    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:39:09.307335 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:09.307287    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" event={"ID":"d7e4b380-d10c-4d5d-b399-7797eb58b804","Type":"ContainerStarted","Data":"8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c"}
Jul 24 00:39:09.307707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:09.307354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" event={"ID":"d7e4b380-d10c-4d5d-b399-7797eb58b804","Type":"ContainerStarted","Data":"f4711f8f60ef59138bbae2d063045c8a0c2d4b3b35446d0548b51bdc15f2b627"}
Jul 24 00:39:09.328654 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:09.328608    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" podStartSLOduration=2.328593973 podStartE2EDuration="2.328593973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:39:09.326788374 +0000 UTC m=+1712.175107116" watchObservedRunningTime="2026-07-24 00:39:09.328593973 +0000 UTC m=+1712.176912714"
Jul 24 00:39:10.207875 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.207844    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:10.212367 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.212342    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:10.311442 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.311415    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:10.312560 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.312537    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:10.874062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.874028    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"]
Jul 24 00:39:10.877122 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.877107    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:10.879586 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.879559    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 00:39:10.879714 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.879599    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 00:39:10.880540 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.880521    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8405249f-serving-cert\""
Jul 24 00:39:10.880635 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.880528    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nv7dj\""
Jul 24 00:39:10.887032 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.887008    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"]
Jul 24 00:39:10.955776 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.955746    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e205b091-3fe8-4c2d-8929-39801b04bd8c-maas-api-tls\") pod \"maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:10.955953 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.955789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e205b091-3fe8-4c2d-8929-39801b04bd8c-openshift-service-ca\") pod \"maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:10.955953 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:10.955870    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5wr\" (UniqueName: \"kubernetes.io/projected/e205b091-3fe8-4c2d-8929-39801b04bd8c-kube-api-access-cr5wr\") pod \"maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:11.057050 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.057014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e205b091-3fe8-4c2d-8929-39801b04bd8c-maas-api-tls\") pod \"maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:11.057244 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.057066    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e205b091-3fe8-4c2d-8929-39801b04bd8c-openshift-service-ca\") pod \"maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:11.057244 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.057129    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5wr\" (UniqueName: \"kubernetes.io/projected/e205b091-3fe8-4c2d-8929-39801b04bd8c-kube-api-access-cr5wr\") pod \"maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:11.057244 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:39:11.057170    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-serving-cert: secret "maas-api-e2e-migrate-8405249f-serving-cert" not found
Jul 24 00:39:11.057431 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:39:11.057268    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e205b091-3fe8-4c2d-8929-39801b04bd8c-maas-api-tls podName:e205b091-3fe8-4c2d-8929-39801b04bd8c nodeName:}" failed. No retries permitted until 2026-07-24 00:39:11.557247317 +0000 UTC m=+1714.405566041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e205b091-3fe8-4c2d-8929-39801b04bd8c-maas-api-tls") pod "maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2" (UID: "e205b091-3fe8-4c2d-8929-39801b04bd8c") : secret "maas-api-e2e-migrate-8405249f-serving-cert" not found
Jul 24 00:39:11.057760 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.057702    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e205b091-3fe8-4c2d-8929-39801b04bd8c-openshift-service-ca\") pod \"maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:11.065932 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.065912    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5wr\" (UniqueName: \"kubernetes.io/projected/e205b091-3fe8-4c2d-8929-39801b04bd8c-kube-api-access-cr5wr\") pod \"maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:11.562694 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.562659    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e205b091-3fe8-4c2d-8929-39801b04bd8c-maas-api-tls\") pod \"maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:11.565118 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.565091    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e205b091-3fe8-4c2d-8929-39801b04bd8c-maas-api-tls\") pod \"maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:11.624916 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.624882    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"]
Jul 24 00:39:11.628139 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.628123    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:11.630555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.630531    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8405249f\""
Jul 24 00:39:11.630690 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.630583    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8405249f-dockercfg-2hs88\""
Jul 24 00:39:11.639013 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.638986    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"]
Jul 24 00:39:11.663932 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.663906    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce8b29b-40f3-4888-a9f4-a894e79c9939-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7\" (UID: \"8ce8b29b-40f3-4888-a9f4-a894e79c9939\") " pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:11.664063 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.663937    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vzm\" (UniqueName: \"kubernetes.io/projected/8ce8b29b-40f3-4888-a9f4-a894e79c9939-kube-api-access-p6vzm\") pod \"payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7\" (UID: \"8ce8b29b-40f3-4888-a9f4-a894e79c9939\") " pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:11.765075 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.765042    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce8b29b-40f3-4888-a9f4-a894e79c9939-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7\" (UID: \"8ce8b29b-40f3-4888-a9f4-a894e79c9939\") " pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:11.765075 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.765074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vzm\" (UniqueName: \"kubernetes.io/projected/8ce8b29b-40f3-4888-a9f4-a894e79c9939-kube-api-access-p6vzm\") pod \"payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7\" (UID: \"8ce8b29b-40f3-4888-a9f4-a894e79c9939\") " pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:11.765798 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.765772    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce8b29b-40f3-4888-a9f4-a894e79c9939-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7\" (UID: \"8ce8b29b-40f3-4888-a9f4-a894e79c9939\") " pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:11.775130 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.775107    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vzm\" (UniqueName: \"kubernetes.io/projected/8ce8b29b-40f3-4888-a9f4-a894e79c9939-kube-api-access-p6vzm\") pod \"payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7\" (UID: \"8ce8b29b-40f3-4888-a9f4-a894e79c9939\") " pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:11.787958 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.787938    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:11.916590 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.916565    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"]
Jul 24 00:39:11.918638 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:39:11.918610    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode205b091_3fe8_4c2d_8929_39801b04bd8c.slice/crio-fcd8da8a2f74bf4b50d8039b28f37ddcae23cd98b485814146edca49c0f0fdd4 WatchSource:0}: Error finding container fcd8da8a2f74bf4b50d8039b28f37ddcae23cd98b485814146edca49c0f0fdd4: Status 404 returned error can't find the container with id fcd8da8a2f74bf4b50d8039b28f37ddcae23cd98b485814146edca49c0f0fdd4
Jul 24 00:39:11.938445 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:11.938419    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:12.064125 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:12.064026    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"]
Jul 24 00:39:12.066189 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:39:12.066164    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce8b29b_40f3_4888_a9f4_a894e79c9939.slice/crio-a6cfe1ac3db9d477ab87fae6f80b3a6d76bf22dfc96da34e093c68559cbb8855 WatchSource:0}: Error finding container a6cfe1ac3db9d477ab87fae6f80b3a6d76bf22dfc96da34e093c68559cbb8855: Status 404 returned error can't find the container with id a6cfe1ac3db9d477ab87fae6f80b3a6d76bf22dfc96da34e093c68559cbb8855
Jul 24 00:39:12.322774 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:12.322744    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2" event={"ID":"e205b091-3fe8-4c2d-8929-39801b04bd8c","Type":"ContainerStarted","Data":"fcd8da8a2f74bf4b50d8039b28f37ddcae23cd98b485814146edca49c0f0fdd4"}
Jul 24 00:39:12.324164 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:12.324140    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7" event={"ID":"8ce8b29b-40f3-4888-a9f4-a894e79c9939","Type":"ContainerStarted","Data":"cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077"}
Jul 24 00:39:12.324260 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:12.324190    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7" event={"ID":"8ce8b29b-40f3-4888-a9f4-a894e79c9939","Type":"ContainerStarted","Data":"a6cfe1ac3db9d477ab87fae6f80b3a6d76bf22dfc96da34e093c68559cbb8855"}
Jul 24 00:39:12.324350 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:12.324337    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:12.354562 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:12.354478    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7" podStartSLOduration=1.354462439 podStartE2EDuration="1.354462439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:39:12.352893153 +0000 UTC m=+1715.201211895" watchObservedRunningTime="2026-07-24 00:39:12.354462439 +0000 UTC m=+1715.202781180"
Jul 24 00:39:13.329648 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:13.329612    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2" event={"ID":"e205b091-3fe8-4c2d-8929-39801b04bd8c","Type":"ContainerStarted","Data":"ae8acbfc9b38f31be663f19e183026d2bab4c9a1a138c3925f3a95a1c84e6d57"}
Jul 24 00:39:13.330035 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:13.329746    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:13.346265 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:13.346216    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2" podStartSLOduration=2.993882015 podStartE2EDuration="3.346201815s" totalImagesPullingTime="352.3198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:39:13.345351605 +0000 UTC m=+1716.193670345" watchObservedRunningTime="2026-07-24 00:39:13.346201815 +0000 UTC m=+1716.194520557"
Jul 24 00:39:19.339520 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:19.339492    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:23.332009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:23.331980    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:25.651493 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:25.650234    2574 patch_prober.go:28] interesting pod/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:35354->10.133.0.33:15021: read: connection reset by peer" start-of-body=
Jul 24 00:39:25.651493 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:25.650299    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:35354->10.133.0.33:15021: read: connection reset by peer"
Jul 24 00:39:26.393990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:26.393957    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerID="8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c" exitCode=0
Jul 24 00:39:26.394154 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:26.394030    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" event={"ID":"d7e4b380-d10c-4d5d-b399-7797eb58b804","Type":"ContainerDied","Data":"8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c"}
Jul 24 00:39:26.394404 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:26.394383    2574 scope.go:117] "RemoveContainer" containerID="8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c"
Jul 24 00:39:26.395346 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:26.395308    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:39:26.395412 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:26.395397    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:39:26.395447 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:26.395430    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:39:27.405914 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:27.405877    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" event={"ID":"d7e4b380-d10c-4d5d-b399-7797eb58b804","Type":"ContainerStarted","Data":"d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77"}
Jul 24 00:39:28.207120 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:28.207080    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:28.207368 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:28.207347    2574 patch_prober.go:28] interesting pod/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 24 00:39:28.207416 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:28.207398    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 24 00:39:29.207355 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:29.207293    2574 patch_prober.go:28] interesting pod/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 24 00:39:29.207742 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:29.207375    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 24 00:39:30.207793 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:30.207751    2574 patch_prober.go:28] interesting pod/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 24 00:39:30.208178 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:30.207826    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 24 00:39:31.207245 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:31.207208    2574 patch_prober.go:28] interesting pod/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 24 00:39:31.207446 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:31.207266    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 24 00:39:32.211022 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:32.210980    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:32.211539 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:32.211514    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:32.212205 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:32.212186    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:46.354993 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.354963    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"]
Jul 24 00:39:46.355439 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.355176    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2" podUID="e205b091-3fe8-4c2d-8929-39801b04bd8c" containerName="maas-api" containerID="cri-o://ae8acbfc9b38f31be663f19e183026d2bab4c9a1a138c3925f3a95a1c84e6d57" gracePeriod=30
Jul 24 00:39:46.418900 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.418871    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"]
Jul 24 00:39:46.419089 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.419066    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7" podUID="8ce8b29b-40f3-4888-a9f4-a894e79c9939" containerName="payload-processing" containerID="cri-o://cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077" gracePeriod=30
Jul 24 00:39:46.485286 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.485246    2574 generic.go:356] "Generic (PLEG): container finished" podID="e205b091-3fe8-4c2d-8929-39801b04bd8c" containerID="ae8acbfc9b38f31be663f19e183026d2bab4c9a1a138c3925f3a95a1c84e6d57" exitCode=0
Jul 24 00:39:46.485453 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.485283    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2" event={"ID":"e205b091-3fe8-4c2d-8929-39801b04bd8c","Type":"ContainerDied","Data":"ae8acbfc9b38f31be663f19e183026d2bab4c9a1a138c3925f3a95a1c84e6d57"}
Jul 24 00:39:46.658801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.658777    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:46.691957 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.691933    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:46.775905 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.775878    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr5wr\" (UniqueName: \"kubernetes.io/projected/e205b091-3fe8-4c2d-8929-39801b04bd8c-kube-api-access-cr5wr\") pod \"e205b091-3fe8-4c2d-8929-39801b04bd8c\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") "
Jul 24 00:39:46.775905 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.775909    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce8b29b-40f3-4888-a9f4-a894e79c9939-plugins-config-volume\") pod \"8ce8b29b-40f3-4888-a9f4-a894e79c9939\" (UID: \"8ce8b29b-40f3-4888-a9f4-a894e79c9939\") "
Jul 24 00:39:46.776145 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.775954    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e205b091-3fe8-4c2d-8929-39801b04bd8c-maas-api-tls\") pod \"e205b091-3fe8-4c2d-8929-39801b04bd8c\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") "
Jul 24 00:39:46.776145 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.775970    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6vzm\" (UniqueName: \"kubernetes.io/projected/8ce8b29b-40f3-4888-a9f4-a894e79c9939-kube-api-access-p6vzm\") pod \"8ce8b29b-40f3-4888-a9f4-a894e79c9939\" (UID: \"8ce8b29b-40f3-4888-a9f4-a894e79c9939\") "
Jul 24 00:39:46.776145 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.775988    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e205b091-3fe8-4c2d-8929-39801b04bd8c-openshift-service-ca\") pod \"e205b091-3fe8-4c2d-8929-39801b04bd8c\" (UID: \"e205b091-3fe8-4c2d-8929-39801b04bd8c\") "
Jul 24 00:39:46.776413 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.776381    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce8b29b-40f3-4888-a9f4-a894e79c9939-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ce8b29b-40f3-4888-a9f4-a894e79c9939" (UID: "8ce8b29b-40f3-4888-a9f4-a894e79c9939"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:39:46.776529 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.776453    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e205b091-3fe8-4c2d-8929-39801b04bd8c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e205b091-3fe8-4c2d-8929-39801b04bd8c" (UID: "e205b091-3fe8-4c2d-8929-39801b04bd8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:39:46.778067 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.778043    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce8b29b-40f3-4888-a9f4-a894e79c9939-kube-api-access-p6vzm" (OuterVolumeSpecName: "kube-api-access-p6vzm") pod "8ce8b29b-40f3-4888-a9f4-a894e79c9939" (UID: "8ce8b29b-40f3-4888-a9f4-a894e79c9939"). InnerVolumeSpecName "kube-api-access-p6vzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:39:46.778158 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.778066    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e205b091-3fe8-4c2d-8929-39801b04bd8c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e205b091-3fe8-4c2d-8929-39801b04bd8c" (UID: "e205b091-3fe8-4c2d-8929-39801b04bd8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:39:46.778158 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.778111    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e205b091-3fe8-4c2d-8929-39801b04bd8c-kube-api-access-cr5wr" (OuterVolumeSpecName: "kube-api-access-cr5wr") pod "e205b091-3fe8-4c2d-8929-39801b04bd8c" (UID: "e205b091-3fe8-4c2d-8929-39801b04bd8c"). InnerVolumeSpecName "kube-api-access-cr5wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:39:46.877571 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.877495    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e205b091-3fe8-4c2d-8929-39801b04bd8c-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:46.877571 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.877523    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6vzm\" (UniqueName: \"kubernetes.io/projected/8ce8b29b-40f3-4888-a9f4-a894e79c9939-kube-api-access-p6vzm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:46.877571 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.877533    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e205b091-3fe8-4c2d-8929-39801b04bd8c-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:46.877571 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.877542    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr5wr\" (UniqueName: \"kubernetes.io/projected/e205b091-3fe8-4c2d-8929-39801b04bd8c-kube-api-access-cr5wr\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:46.877571 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:46.877550    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce8b29b-40f3-4888-a9f4-a894e79c9939-plugins-config-volume\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:47.490301 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.490267    2574 generic.go:356] "Generic (PLEG): container finished" podID="8ce8b29b-40f3-4888-a9f4-a894e79c9939" containerID="cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077" exitCode=0
Jul 24 00:39:47.490768 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.490334    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"
Jul 24 00:39:47.490768 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.490354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7" event={"ID":"8ce8b29b-40f3-4888-a9f4-a894e79c9939","Type":"ContainerDied","Data":"cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077"}
Jul 24 00:39:47.490768 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.490404    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7" event={"ID":"8ce8b29b-40f3-4888-a9f4-a894e79c9939","Type":"ContainerDied","Data":"a6cfe1ac3db9d477ab87fae6f80b3a6d76bf22dfc96da34e093c68559cbb8855"}
Jul 24 00:39:47.490768 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.490435    2574 scope.go:117] "RemoveContainer" containerID="cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077"
Jul 24 00:39:47.491800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.491785    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"
Jul 24 00:39:47.491902 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.491795    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2" event={"ID":"e205b091-3fe8-4c2d-8929-39801b04bd8c","Type":"ContainerDied","Data":"fcd8da8a2f74bf4b50d8039b28f37ddcae23cd98b485814146edca49c0f0fdd4"}
Jul 24 00:39:47.499835 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.499817    2574 scope.go:117] "RemoveContainer" containerID="cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077"
Jul 24 00:39:47.500106 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:39:47.500084    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077\": container with ID starting with cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077 not found: ID does not exist" containerID="cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077"
Jul 24 00:39:47.500196 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.500113    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077"} err="failed to get container status \"cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077\": rpc error: code = NotFound desc = could not find container \"cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077\": container with ID starting with cddee99261f1228eae033534d503e2856b3ec3cccaba24ba6cfff02011696077 not found: ID does not exist"
Jul 24 00:39:47.500196 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.500135    2574 scope.go:117] "RemoveContainer" containerID="ae8acbfc9b38f31be663f19e183026d2bab4c9a1a138c3925f3a95a1c84e6d57"
Jul 24 00:39:47.515456 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.515435    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"]
Jul 24 00:39:47.517761 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.517737    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8405249f-7df5fbcd97-2w8c7"]
Jul 24 00:39:47.528057 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.528037    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"]
Jul 24 00:39:47.530883 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.530864    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8405249f-75bb7b5cd8-5gch2"]
Jul 24 00:39:47.752675 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.752597    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce8b29b-40f3-4888-a9f4-a894e79c9939" path="/var/lib/kubelet/pods/8ce8b29b-40f3-4888-a9f4-a894e79c9939/volumes"
Jul 24 00:39:47.752978 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:47.752964    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e205b091-3fe8-4c2d-8929-39801b04bd8c" path="/var/lib/kubelet/pods/e205b091-3fe8-4c2d-8929-39801b04bd8c/volumes"
Jul 24 00:39:51.094615 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:51.094580    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"]
Jul 24 00:39:51.095011 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:51.094828    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy" containerID="cri-o://d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77" gracePeriod=30
Jul 24 00:39:54.466780 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.466749    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"]
Jul 24 00:39:54.467133 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.467026    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce8b29b-40f3-4888-a9f4-a894e79c9939" containerName="payload-processing"
Jul 24 00:39:54.467133 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.467037    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce8b29b-40f3-4888-a9f4-a894e79c9939" containerName="payload-processing"
Jul 24 00:39:54.467133 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.467047    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e205b091-3fe8-4c2d-8929-39801b04bd8c" containerName="maas-api"
Jul 24 00:39:54.467133 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.467052    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e205b091-3fe8-4c2d-8929-39801b04bd8c" containerName="maas-api"
Jul 24 00:39:54.467133 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.467111    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e205b091-3fe8-4c2d-8929-39801b04bd8c" containerName="maas-api"
Jul 24 00:39:54.467133 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.467121    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce8b29b-40f3-4888-a9f4-a894e79c9939" containerName="payload-processing"
Jul 24 00:39:54.471652 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.471630    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:54.474079 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.474059    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nv7dj\""
Jul 24 00:39:54.475144 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.475125    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dd068a7a-serving-cert\""
Jul 24 00:39:54.475249 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.475180    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 00:39:54.475334 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.475249    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 00:39:54.480167 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.480146    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"]
Jul 24 00:39:54.545241 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.545211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d64d936-bbe0-4fed-82eb-f435147a7471-maas-api-tls\") pod \"maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:54.545241 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.545239    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z69l\" (UniqueName: \"kubernetes.io/projected/1d64d936-bbe0-4fed-82eb-f435147a7471-kube-api-access-6z69l\") pod \"maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:54.545477 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.545382    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d64d936-bbe0-4fed-82eb-f435147a7471-openshift-service-ca\") pod \"maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:54.645922 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.645888    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d64d936-bbe0-4fed-82eb-f435147a7471-openshift-service-ca\") pod \"maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:54.646095 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.645930    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d64d936-bbe0-4fed-82eb-f435147a7471-maas-api-tls\") pod \"maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:54.646095 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.645949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z69l\" (UniqueName: \"kubernetes.io/projected/1d64d936-bbe0-4fed-82eb-f435147a7471-kube-api-access-6z69l\") pod \"maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:54.646095 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:39:54.646044    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-serving-cert: secret "maas-api-e2e-ait-dd068a7a-serving-cert" not found
Jul 24 00:39:54.646271 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:39:54.646117    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d64d936-bbe0-4fed-82eb-f435147a7471-maas-api-tls podName:1d64d936-bbe0-4fed-82eb-f435147a7471 nodeName:}" failed. No retries permitted until 2026-07-24 00:39:55.146099406 +0000 UTC m=+1757.994418125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d64d936-bbe0-4fed-82eb-f435147a7471-maas-api-tls") pod "maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl" (UID: "1d64d936-bbe0-4fed-82eb-f435147a7471") : secret "maas-api-e2e-ait-dd068a7a-serving-cert" not found
Jul 24 00:39:54.646655 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.646631    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d64d936-bbe0-4fed-82eb-f435147a7471-openshift-service-ca\") pod \"maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:54.655177 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:54.655155    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z69l\" (UniqueName: \"kubernetes.io/projected/1d64d936-bbe0-4fed-82eb-f435147a7471-kube-api-access-6z69l\") pod \"maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:55.075650 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.075613    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"]
Jul 24 00:39:55.078889 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.078873    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:39:55.081468 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.081445    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dd068a7a\""
Jul 24 00:39:55.081559 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.081453    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dd068a7a-dockercfg-qx2wl\""
Jul 24 00:39:55.086919 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.086900    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"]
Jul 24 00:39:55.089890 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.089871    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"]
Jul 24 00:39:55.092843 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.092823    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:39:55.104869 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.104848    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"]
Jul 24 00:39:55.149516 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.149492    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d64d936-bbe0-4fed-82eb-f435147a7471-maas-api-tls\") pod \"maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:55.149622 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.149538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7tw9\" (UniqueName: \"kubernetes.io/projected/f8e46bbc-a31c-4008-8a73-8213f1912b8e-kube-api-access-b7tw9\") pod \"payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt\" (UID: \"f8e46bbc-a31c-4008-8a73-8213f1912b8e\") " pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:39:55.149622 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.149584    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z85k\" (UniqueName: \"kubernetes.io/projected/f722a27d-c312-43dc-94d3-7ea88eb639a0-kube-api-access-2z85k\") pod \"payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk\" (UID: \"f722a27d-c312-43dc-94d3-7ea88eb639a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:39:55.149622 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.149611    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e46bbc-a31c-4008-8a73-8213f1912b8e-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt\" (UID: \"f8e46bbc-a31c-4008-8a73-8213f1912b8e\") " pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:39:55.149727 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.149647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f722a27d-c312-43dc-94d3-7ea88eb639a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk\" (UID: \"f722a27d-c312-43dc-94d3-7ea88eb639a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:39:55.151821 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.151803    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d64d936-bbe0-4fed-82eb-f435147a7471-maas-api-tls\") pod \"maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:55.250087 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.250053    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7tw9\" (UniqueName: \"kubernetes.io/projected/f8e46bbc-a31c-4008-8a73-8213f1912b8e-kube-api-access-b7tw9\") pod \"payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt\" (UID: \"f8e46bbc-a31c-4008-8a73-8213f1912b8e\") " pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:39:55.250237 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.250099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z85k\" (UniqueName: \"kubernetes.io/projected/f722a27d-c312-43dc-94d3-7ea88eb639a0-kube-api-access-2z85k\") pod \"payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk\" (UID: \"f722a27d-c312-43dc-94d3-7ea88eb639a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:39:55.250237 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.250125    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e46bbc-a31c-4008-8a73-8213f1912b8e-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt\" (UID: \"f8e46bbc-a31c-4008-8a73-8213f1912b8e\") " pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:39:55.250237 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.250157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f722a27d-c312-43dc-94d3-7ea88eb639a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk\" (UID: \"f722a27d-c312-43dc-94d3-7ea88eb639a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:39:55.250881 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.250863    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f722a27d-c312-43dc-94d3-7ea88eb639a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk\" (UID: \"f722a27d-c312-43dc-94d3-7ea88eb639a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:39:55.250924 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.250865    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e46bbc-a31c-4008-8a73-8213f1912b8e-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt\" (UID: \"f8e46bbc-a31c-4008-8a73-8213f1912b8e\") " pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:39:55.258309 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.258281    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7tw9\" (UniqueName: \"kubernetes.io/projected/f8e46bbc-a31c-4008-8a73-8213f1912b8e-kube-api-access-b7tw9\") pod \"payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt\" (UID: \"f8e46bbc-a31c-4008-8a73-8213f1912b8e\") " pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:39:55.258460 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.258436    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z85k\" (UniqueName: \"kubernetes.io/projected/f722a27d-c312-43dc-94d3-7ea88eb639a0-kube-api-access-2z85k\") pod \"payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk\" (UID: \"f722a27d-c312-43dc-94d3-7ea88eb639a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:39:55.383630 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.383547    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:55.390659 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.390639    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:39:55.402619 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.402594    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:39:55.535345 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.535298    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"]
Jul 24 00:39:55.538828 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:39:55.538802    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d64d936_bbe0_4fed_82eb_f435147a7471.slice/crio-fd2236a0285b7f8bdee0282cf010e6e6d6df209782eae4c46344ebc3860f318b WatchSource:0}: Error finding container fd2236a0285b7f8bdee0282cf010e6e6d6df209782eae4c46344ebc3860f318b: Status 404 returned error can't find the container with id fd2236a0285b7f8bdee0282cf010e6e6d6df209782eae4c46344ebc3860f318b
Jul 24 00:39:55.762690 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.762667    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"]
Jul 24 00:39:55.765122 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:39:55.765091    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf722a27d_c312_43dc_94d3_7ea88eb639a0.slice/crio-69206d010256233050c54c0bfaa9c321117ff1c5a8beb3ed322a920565dd6118 WatchSource:0}: Error finding container 69206d010256233050c54c0bfaa9c321117ff1c5a8beb3ed322a920565dd6118: Status 404 returned error can't find the container with id 69206d010256233050c54c0bfaa9c321117ff1c5a8beb3ed322a920565dd6118
Jul 24 00:39:55.766258 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:55.766228    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"]
Jul 24 00:39:55.768064 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:39:55.768042    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e46bbc_a31c_4008_8a73_8213f1912b8e.slice/crio-a5703d5aeb4ab44ae31cde465d381a7ae8e7788a1519e50e32540ca96a70d4d0 WatchSource:0}: Error finding container a5703d5aeb4ab44ae31cde465d381a7ae8e7788a1519e50e32540ca96a70d4d0: Status 404 returned error can't find the container with id a5703d5aeb4ab44ae31cde465d381a7ae8e7788a1519e50e32540ca96a70d4d0
Jul 24 00:39:56.337605 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.337579    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:56.358099 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358072    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-credential-socket\") pod \"d7e4b380-d10c-4d5d-b399-7797eb58b804\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") "
Jul 24 00:39:56.358255 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358112    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-token\") pod \"d7e4b380-d10c-4d5d-b399-7797eb58b804\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") "
Jul 24 00:39:56.358255 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358132    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e4b380-d10c-4d5d-b399-7797eb58b804-istiod-ca-cert\") pod \"d7e4b380-d10c-4d5d-b399-7797eb58b804\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") "
Jul 24 00:39:56.358255 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358196    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-certs\") pod \"d7e4b380-d10c-4d5d-b399-7797eb58b804\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") "
Jul 24 00:39:56.358255 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358234    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-socket\") pod \"d7e4b380-d10c-4d5d-b399-7797eb58b804\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") "
Jul 24 00:39:56.358491 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358277    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-data\") pod \"d7e4b380-d10c-4d5d-b399-7797eb58b804\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") "
Jul 24 00:39:56.358491 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358324    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-envoy\") pod \"d7e4b380-d10c-4d5d-b399-7797eb58b804\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") "
Jul 24 00:39:56.358491 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358365    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkqfx\" (UniqueName: \"kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-kube-api-access-qkqfx\") pod \"d7e4b380-d10c-4d5d-b399-7797eb58b804\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") "
Jul 24 00:39:56.358491 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358394    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-podinfo\") pod \"d7e4b380-d10c-4d5d-b399-7797eb58b804\" (UID: \"d7e4b380-d10c-4d5d-b399-7797eb58b804\") "
Jul 24 00:39:56.358724 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358690    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7e4b380-d10c-4d5d-b399-7797eb58b804" (UID: "d7e4b380-d10c-4d5d-b399-7797eb58b804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:39:56.358795 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358705    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7e4b380-d10c-4d5d-b399-7797eb58b804" (UID: "d7e4b380-d10c-4d5d-b399-7797eb58b804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:39:56.359019 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.358964    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e4b380-d10c-4d5d-b399-7797eb58b804-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7e4b380-d10c-4d5d-b399-7797eb58b804" (UID: "d7e4b380-d10c-4d5d-b399-7797eb58b804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:39:56.359262 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.359177    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7e4b380-d10c-4d5d-b399-7797eb58b804" (UID: "d7e4b380-d10c-4d5d-b399-7797eb58b804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:39:56.360131 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.360047    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7e4b380-d10c-4d5d-b399-7797eb58b804" (UID: "d7e4b380-d10c-4d5d-b399-7797eb58b804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:39:56.361075 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.361042    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7e4b380-d10c-4d5d-b399-7797eb58b804" (UID: "d7e4b380-d10c-4d5d-b399-7797eb58b804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 00:39:56.361464 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.361438    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7e4b380-d10c-4d5d-b399-7797eb58b804" (UID: "d7e4b380-d10c-4d5d-b399-7797eb58b804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:39:56.361714 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.361686    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-kube-api-access-qkqfx" (OuterVolumeSpecName: "kube-api-access-qkqfx") pod "d7e4b380-d10c-4d5d-b399-7797eb58b804" (UID: "d7e4b380-d10c-4d5d-b399-7797eb58b804"). InnerVolumeSpecName "kube-api-access-qkqfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:39:56.362186 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.362161    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7e4b380-d10c-4d5d-b399-7797eb58b804" (UID: "d7e4b380-d10c-4d5d-b399-7797eb58b804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:39:56.459847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.459819    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:56.459847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.459844    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-workload-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:56.459847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.459852    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-data\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:56.460049 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.459860    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-envoy\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:56.460049 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.459868    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkqfx\" (UniqueName: \"kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-kube-api-access-qkqfx\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:56.460049 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.459876    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-podinfo\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:56.460049 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.459885    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e4b380-d10c-4d5d-b399-7797eb58b804-credential-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:56.460049 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.459893    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e4b380-d10c-4d5d-b399-7797eb58b804-istio-token\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:56.460049 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.459900    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e4b380-d10c-4d5d-b399-7797eb58b804-istiod-ca-cert\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:39:56.536476 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.536446    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt" event={"ID":"f8e46bbc-a31c-4008-8a73-8213f1912b8e","Type":"ContainerStarted","Data":"e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3"}
Jul 24 00:39:56.536833 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.536496    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt" event={"ID":"f8e46bbc-a31c-4008-8a73-8213f1912b8e","Type":"ContainerStarted","Data":"a5703d5aeb4ab44ae31cde465d381a7ae8e7788a1519e50e32540ca96a70d4d0"}
Jul 24 00:39:56.536833 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.536561    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:39:56.537955 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.537930    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk" event={"ID":"f722a27d-c312-43dc-94d3-7ea88eb639a0","Type":"ContainerStarted","Data":"0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a"}
Jul 24 00:39:56.538072 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.537967    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk" event={"ID":"f722a27d-c312-43dc-94d3-7ea88eb639a0","Type":"ContainerStarted","Data":"69206d010256233050c54c0bfaa9c321117ff1c5a8beb3ed322a920565dd6118"}
Jul 24 00:39:56.538072 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.537993    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:39:56.539453 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.539426    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerID="d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77" exitCode=0
Jul 24 00:39:56.539561 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.539475    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"
Jul 24 00:39:56.539561 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.539521    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" event={"ID":"d7e4b380-d10c-4d5d-b399-7797eb58b804","Type":"ContainerDied","Data":"d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77"}
Jul 24 00:39:56.539666 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.539598    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx" event={"ID":"d7e4b380-d10c-4d5d-b399-7797eb58b804","Type":"ContainerDied","Data":"f4711f8f60ef59138bbae2d063045c8a0c2d4b3b35446d0548b51bdc15f2b627"}
Jul 24 00:39:56.539666 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.539632    2574 scope.go:117] "RemoveContainer" containerID="d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77"
Jul 24 00:39:56.541444 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.541420    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl" event={"ID":"1d64d936-bbe0-4fed-82eb-f435147a7471","Type":"ContainerStarted","Data":"ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7"}
Jul 24 00:39:56.541553 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.541465    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl" event={"ID":"1d64d936-bbe0-4fed-82eb-f435147a7471","Type":"ContainerStarted","Data":"fd2236a0285b7f8bdee0282cf010e6e6d6df209782eae4c46344ebc3860f318b"}
Jul 24 00:39:56.541607 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.541567    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:39:56.551401 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.551382    2574 scope.go:117] "RemoveContainer" containerID="8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c"
Jul 24 00:39:56.558101 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.558053    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt" podStartSLOduration=1.558035613 podStartE2EDuration="1.558035613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:39:56.555889795 +0000 UTC m=+1759.404208537" watchObservedRunningTime="2026-07-24 00:39:56.558035613 +0000 UTC m=+1759.406354357"
Jul 24 00:39:56.560973 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.560946    2574 scope.go:117] "RemoveContainer" containerID="d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77"
Jul 24 00:39:56.561363 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:39:56.561336    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77\": container with ID starting with d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77 not found: ID does not exist" containerID="d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77"
Jul 24 00:39:56.561453 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.561372    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77"} err="failed to get container status \"d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77\": rpc error: code = NotFound desc = could not find container \"d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77\": container with ID starting with d652710cdf7fde3e24e68b6a6330a608a68878d595c004e14909c4ae653e3f77 not found: ID does not exist"
Jul 24 00:39:56.561453 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.561395    2574 scope.go:117] "RemoveContainer" containerID="8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c"
Jul 24 00:39:56.561805 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:39:56.561780    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c\": container with ID starting with 8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c not found: ID does not exist" containerID="8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c"
Jul 24 00:39:56.561891 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.561814    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c"} err="failed to get container status \"8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c\": rpc error: code = NotFound desc = could not find container \"8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c\": container with ID starting with 8249b8bfb230ba253f772a12a22a90e5a9262b37c2fc9268345d15eb92a4471c not found: ID does not exist"
Jul 24 00:39:56.568801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.568779    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"]
Jul 24 00:39:56.572923 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.572903    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8405249f-gw-openshift-default-6ccdb6ffd4-5g6jx"]
Jul 24 00:39:56.587154 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.587087    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk" podStartSLOduration=1.587072182 podStartE2EDuration="1.587072182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:39:56.586303088 +0000 UTC m=+1759.434621831" watchObservedRunningTime="2026-07-24 00:39:56.587072182 +0000 UTC m=+1759.435390926"
Jul 24 00:39:56.602100 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:56.602060    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl" podStartSLOduration=2.216620494 podStartE2EDuration="2.602048784s" totalImagesPullingTime="385.42829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:39:56.60192723 +0000 UTC m=+1759.450245971" watchObservedRunningTime="2026-07-24 00:39:56.602048784 +0000 UTC m=+1759.450367525"
Jul 24 00:39:57.752427 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:39:57.752388    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" path="/var/lib/kubelet/pods/d7e4b380-d10c-4d5d-b399-7797eb58b804/volumes"
Jul 24 00:40:02.554875 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:02.554848    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:40:07.549467 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:07.549431    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:40:07.549821 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:07.549500    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:40:30.736819 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:30.736730    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"]
Jul 24 00:40:30.737412 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:30.736972    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl" podUID="1d64d936-bbe0-4fed-82eb-f435147a7471" containerName="maas-api" containerID="cri-o://ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7" gracePeriod=30
Jul 24 00:40:30.798063 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:30.798030    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"]
Jul 24 00:40:30.798325 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:30.798279    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt" podUID="f8e46bbc-a31c-4008-8a73-8213f1912b8e" containerName="payload-processing" containerID="cri-o://e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3" gracePeriod=30
Jul 24 00:40:30.806785 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:30.806743    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"]
Jul 24 00:40:30.806941 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:30.806923    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk" podUID="f722a27d-c312-43dc-94d3-7ea88eb639a0" containerName="payload-pre-processing" containerID="cri-o://0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a" gracePeriod=30
Jul 24 00:40:31.083759 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.083728    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:40:31.086503 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.086485    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:40:31.089764 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.089745    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:40:31.243218 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.243187    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d64d936-bbe0-4fed-82eb-f435147a7471-maas-api-tls\") pod \"1d64d936-bbe0-4fed-82eb-f435147a7471\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") "
Jul 24 00:40:31.243399 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.243241    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e46bbc-a31c-4008-8a73-8213f1912b8e-plugins-config-volume\") pod \"f8e46bbc-a31c-4008-8a73-8213f1912b8e\" (UID: \"f8e46bbc-a31c-4008-8a73-8213f1912b8e\") "
Jul 24 00:40:31.243399 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.243261    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z85k\" (UniqueName: \"kubernetes.io/projected/f722a27d-c312-43dc-94d3-7ea88eb639a0-kube-api-access-2z85k\") pod \"f722a27d-c312-43dc-94d3-7ea88eb639a0\" (UID: \"f722a27d-c312-43dc-94d3-7ea88eb639a0\") "
Jul 24 00:40:31.243399 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.243285    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z69l\" (UniqueName: \"kubernetes.io/projected/1d64d936-bbe0-4fed-82eb-f435147a7471-kube-api-access-6z69l\") pod \"1d64d936-bbe0-4fed-82eb-f435147a7471\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") "
Jul 24 00:40:31.243399 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.243310    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d64d936-bbe0-4fed-82eb-f435147a7471-openshift-service-ca\") pod \"1d64d936-bbe0-4fed-82eb-f435147a7471\" (UID: \"1d64d936-bbe0-4fed-82eb-f435147a7471\") "
Jul 24 00:40:31.243399 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.243351    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7tw9\" (UniqueName: \"kubernetes.io/projected/f8e46bbc-a31c-4008-8a73-8213f1912b8e-kube-api-access-b7tw9\") pod \"f8e46bbc-a31c-4008-8a73-8213f1912b8e\" (UID: \"f8e46bbc-a31c-4008-8a73-8213f1912b8e\") "
Jul 24 00:40:31.243700 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.243406    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f722a27d-c312-43dc-94d3-7ea88eb639a0-plugins-config-volume\") pod \"f722a27d-c312-43dc-94d3-7ea88eb639a0\" (UID: \"f722a27d-c312-43dc-94d3-7ea88eb639a0\") "
Jul 24 00:40:31.243700 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.243650    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e46bbc-a31c-4008-8a73-8213f1912b8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8e46bbc-a31c-4008-8a73-8213f1912b8e" (UID: "f8e46bbc-a31c-4008-8a73-8213f1912b8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:40:31.243700 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.243672    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d64d936-bbe0-4fed-82eb-f435147a7471-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d64d936-bbe0-4fed-82eb-f435147a7471" (UID: "1d64d936-bbe0-4fed-82eb-f435147a7471"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:40:31.243952 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.243927    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f722a27d-c312-43dc-94d3-7ea88eb639a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f722a27d-c312-43dc-94d3-7ea88eb639a0" (UID: "f722a27d-c312-43dc-94d3-7ea88eb639a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:40:31.245470 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.245444    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f722a27d-c312-43dc-94d3-7ea88eb639a0-kube-api-access-2z85k" (OuterVolumeSpecName: "kube-api-access-2z85k") pod "f722a27d-c312-43dc-94d3-7ea88eb639a0" (UID: "f722a27d-c312-43dc-94d3-7ea88eb639a0"). InnerVolumeSpecName "kube-api-access-2z85k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:40:31.245547 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.245481    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d64d936-bbe0-4fed-82eb-f435147a7471-kube-api-access-6z69l" (OuterVolumeSpecName: "kube-api-access-6z69l") pod "1d64d936-bbe0-4fed-82eb-f435147a7471" (UID: "1d64d936-bbe0-4fed-82eb-f435147a7471"). InnerVolumeSpecName "kube-api-access-6z69l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:40:31.245778 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.245759    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e46bbc-a31c-4008-8a73-8213f1912b8e-kube-api-access-b7tw9" (OuterVolumeSpecName: "kube-api-access-b7tw9") pod "f8e46bbc-a31c-4008-8a73-8213f1912b8e" (UID: "f8e46bbc-a31c-4008-8a73-8213f1912b8e"). InnerVolumeSpecName "kube-api-access-b7tw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:40:31.245778 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.245768    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d64d936-bbe0-4fed-82eb-f435147a7471-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d64d936-bbe0-4fed-82eb-f435147a7471" (UID: "1d64d936-bbe0-4fed-82eb-f435147a7471"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:40:31.344655 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.344630    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d64d936-bbe0-4fed-82eb-f435147a7471-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:40:31.344655 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.344651    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e46bbc-a31c-4008-8a73-8213f1912b8e-plugins-config-volume\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:40:31.344794 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.344663    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z85k\" (UniqueName: \"kubernetes.io/projected/f722a27d-c312-43dc-94d3-7ea88eb639a0-kube-api-access-2z85k\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:40:31.344794 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.344672    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z69l\" (UniqueName: \"kubernetes.io/projected/1d64d936-bbe0-4fed-82eb-f435147a7471-kube-api-access-6z69l\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:40:31.344794 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.344680    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d64d936-bbe0-4fed-82eb-f435147a7471-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:40:31.344794 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.344688    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7tw9\" (UniqueName: \"kubernetes.io/projected/f8e46bbc-a31c-4008-8a73-8213f1912b8e-kube-api-access-b7tw9\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:40:31.344794 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.344697    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f722a27d-c312-43dc-94d3-7ea88eb639a0-plugins-config-volume\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:40:31.693626 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.693532    2574 generic.go:356] "Generic (PLEG): container finished" podID="f8e46bbc-a31c-4008-8a73-8213f1912b8e" containerID="e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3" exitCode=0
Jul 24 00:40:31.693626 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.693580    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"
Jul 24 00:40:31.693626 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.693607    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt" event={"ID":"f8e46bbc-a31c-4008-8a73-8213f1912b8e","Type":"ContainerDied","Data":"e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3"}
Jul 24 00:40:31.693893 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.693657    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt" event={"ID":"f8e46bbc-a31c-4008-8a73-8213f1912b8e","Type":"ContainerDied","Data":"a5703d5aeb4ab44ae31cde465d381a7ae8e7788a1519e50e32540ca96a70d4d0"}
Jul 24 00:40:31.693893 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.693684    2574 scope.go:117] "RemoveContainer" containerID="e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3"
Jul 24 00:40:31.694885 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.694857    2574 generic.go:356] "Generic (PLEG): container finished" podID="f722a27d-c312-43dc-94d3-7ea88eb639a0" containerID="0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a" exitCode=0
Jul 24 00:40:31.695019 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.694897    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"
Jul 24 00:40:31.695019 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.694898    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk" event={"ID":"f722a27d-c312-43dc-94d3-7ea88eb639a0","Type":"ContainerDied","Data":"0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a"}
Jul 24 00:40:31.695138 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.695051    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk" event={"ID":"f722a27d-c312-43dc-94d3-7ea88eb639a0","Type":"ContainerDied","Data":"69206d010256233050c54c0bfaa9c321117ff1c5a8beb3ed322a920565dd6118"}
Jul 24 00:40:31.696191 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.696168    2574 generic.go:356] "Generic (PLEG): container finished" podID="1d64d936-bbe0-4fed-82eb-f435147a7471" containerID="ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7" exitCode=0
Jul 24 00:40:31.696285 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.696193    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl" event={"ID":"1d64d936-bbe0-4fed-82eb-f435147a7471","Type":"ContainerDied","Data":"ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7"}
Jul 24 00:40:31.696285 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.696218    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"
Jul 24 00:40:31.696285 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.696244    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl" event={"ID":"1d64d936-bbe0-4fed-82eb-f435147a7471","Type":"ContainerDied","Data":"fd2236a0285b7f8bdee0282cf010e6e6d6df209782eae4c46344ebc3860f318b"}
Jul 24 00:40:31.712780 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.712761    2574 scope.go:117] "RemoveContainer" containerID="e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3"
Jul 24 00:40:31.713187 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:40:31.713163    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3\": container with ID starting with e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3 not found: ID does not exist" containerID="e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3"
Jul 24 00:40:31.713283 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.713193    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3"} err="failed to get container status \"e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3\": rpc error: code = NotFound desc = could not find container \"e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3\": container with ID starting with e2afaffd70f16665c9a068a730a868012c1c0e0a1c62f65194eaff74b19a23b3 not found: ID does not exist"
Jul 24 00:40:31.713283 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.713210    2574 scope.go:117] "RemoveContainer" containerID="0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a"
Jul 24 00:40:31.720589 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.720572    2574 scope.go:117] "RemoveContainer" containerID="0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a"
Jul 24 00:40:31.720829 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:40:31.720810    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a\": container with ID starting with 0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a not found: ID does not exist" containerID="0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a"
Jul 24 00:40:31.720885 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.720835    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a"} err="failed to get container status \"0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a\": rpc error: code = NotFound desc = could not find container \"0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a\": container with ID starting with 0b1eac99be90c1ebd1250f8caf316d1e03de5e0ba4c4ea0a5a0130bcc066b03a not found: ID does not exist"
Jul 24 00:40:31.720885 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.720850    2574 scope.go:117] "RemoveContainer" containerID="ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7"
Jul 24 00:40:31.724811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.724792    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"]
Jul 24 00:40:31.729004 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.728984    2574 scope.go:117] "RemoveContainer" containerID="ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7"
Jul 24 00:40:31.729279 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:40:31.729253    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7\": container with ID starting with ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7 not found: ID does not exist" containerID="ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7"
Jul 24 00:40:31.729424 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.729285    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7"} err="failed to get container status \"ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7\": rpc error: code = NotFound desc = could not find container \"ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7\": container with ID starting with ec1d28c980dfb3bd08db42582bb61a7cd378a957bafc1708b176ed819d1aa3c7 not found: ID does not exist"
Jul 24 00:40:31.729503 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.729487    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd068a7a-d76f6687d-hm2kl"]
Jul 24 00:40:31.738682 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.738661    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"]
Jul 24 00:40:31.742679 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.742661    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd068a7a-788bddc798-pkprt"]
Jul 24 00:40:31.752440 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.752417    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d64d936-bbe0-4fed-82eb-f435147a7471" path="/var/lib/kubelet/pods/1d64d936-bbe0-4fed-82eb-f435147a7471/volumes"
Jul 24 00:40:31.753254 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.752964    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e46bbc-a31c-4008-8a73-8213f1912b8e" path="/var/lib/kubelet/pods/f8e46bbc-a31c-4008-8a73-8213f1912b8e/volumes"
Jul 24 00:40:31.753751 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.753731    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"]
Jul 24 00:40:31.754916 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:31.754900    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd068a7a-5f8bb69cc6-7fsvk"]
Jul 24 00:40:33.752504 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:33.752471    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f722a27d-c312-43dc-94d3-7ea88eb639a0" path="/var/lib/kubelet/pods/f722a27d-c312-43dc-94d3-7ea88eb639a0/volumes"
Jul 24 00:40:37.363402 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363365    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"]
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363662    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f722a27d-c312-43dc-94d3-7ea88eb639a0" containerName="payload-pre-processing"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363673    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f722a27d-c312-43dc-94d3-7ea88eb639a0" containerName="payload-pre-processing"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363685    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363691    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363704    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e46bbc-a31c-4008-8a73-8213f1912b8e" containerName="payload-processing"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363709    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e46bbc-a31c-4008-8a73-8213f1912b8e" containerName="payload-processing"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363715    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d64d936-bbe0-4fed-82eb-f435147a7471" containerName="maas-api"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363720    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d64d936-bbe0-4fed-82eb-f435147a7471" containerName="maas-api"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363768    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d64d936-bbe0-4fed-82eb-f435147a7471" containerName="maas-api"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363778    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363785    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363790    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e46bbc-a31c-4008-8a73-8213f1912b8e" containerName="payload-processing"
Jul 24 00:40:37.363947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.363796    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f722a27d-c312-43dc-94d3-7ea88eb639a0" containerName="payload-pre-processing"
Jul 24 00:40:37.366734 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.366715    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.369888 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.369863    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-bd6ad2bc-openshift-default-dockercfg-s5fzf\""
Jul 24 00:40:37.379699 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.379677    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"]
Jul 24 00:40:37.496732 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.496697    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qttzk\" (UniqueName: \"kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-kube-api-access-qttzk\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.496928 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.496744    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-data\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.496928 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.496773    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-certs\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.496928 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.496788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99965c9c-338f-4cb0-8277-a261189678a1-istio-podinfo\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.496928 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.496853    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-istio-token\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.497152 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.496932    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-socket\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.497152 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.496959    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-credential-socket\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.497152 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.496997    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99965c9c-338f-4cb0-8277-a261189678a1-istiod-ca-cert\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.497152 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.497062    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-envoy\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.597532 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.597499    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-data\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.597723 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.597546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-certs\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.597723 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.597569    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99965c9c-338f-4cb0-8277-a261189678a1-istio-podinfo\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.597723 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.597593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-istio-token\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.597723 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.597637    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-socket\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.597723 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.597664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-credential-socket\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.597723 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.597694    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99965c9c-338f-4cb0-8277-a261189678a1-istiod-ca-cert\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.598029 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.597741    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-envoy\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.598029 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.597778    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qttzk\" (UniqueName: \"kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-kube-api-access-qttzk\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.598029 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.598011    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-certs\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.598169 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.598073    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-socket\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.598169 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.598130    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-credential-socket\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.598247 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.598181    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-data\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.598609 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.598585    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99965c9c-338f-4cb0-8277-a261189678a1-istiod-ca-cert\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.600249 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.600222    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-envoy\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.600462 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.600446    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99965c9c-338f-4cb0-8277-a261189678a1-istio-podinfo\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.605585 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.605563    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qttzk\" (UniqueName: \"kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-kube-api-access-qttzk\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.605816 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.605795    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-istio-token\") pod \"e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") " pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.679965 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.679891    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-bd6ad2bc-openshift-default-dockercfg-s5fzf\""
Jul 24 00:40:37.687846 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.687823    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:37.829556 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.829530    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"]
Jul 24 00:40:37.831425 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:40:37.831390    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99965c9c_338f_4cb0_8277_a261189678a1.slice/crio-1a0059ff3f22659f7e33649509b32febfce2b4f04e91b7b736265006531dbd57 WatchSource:0}: Error finding container 1a0059ff3f22659f7e33649509b32febfce2b4f04e91b7b736265006531dbd57: Status 404 returned error can't find the container with id 1a0059ff3f22659f7e33649509b32febfce2b4f04e91b7b736265006531dbd57
Jul 24 00:40:37.833610 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.833498    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:40:37.833610 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.833563    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:40:37.833610 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:37.833591    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:40:38.730448 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:38.730413    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" event={"ID":"99965c9c-338f-4cb0-8277-a261189678a1","Type":"ContainerStarted","Data":"cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d"}
Jul 24 00:40:38.730829 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:38.730464    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" event={"ID":"99965c9c-338f-4cb0-8277-a261189678a1","Type":"ContainerStarted","Data":"1a0059ff3f22659f7e33649509b32febfce2b4f04e91b7b736265006531dbd57"}
Jul 24 00:40:38.753917 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:38.753864    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" podStartSLOduration=1.753848984 podStartE2EDuration="1.753848984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:40:38.753049326 +0000 UTC m=+1801.601368069" watchObservedRunningTime="2026-07-24 00:40:38.753848984 +0000 UTC m=+1801.602167724"
Jul 24 00:40:39.688861 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:39.688824    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:39.694038 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:39.694011    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:39.734633 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:39.734604    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:39.735527 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:39.735508    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:40.175071 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.175037    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"]
Jul 24 00:40:40.175347 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.175335    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy"
Jul 24 00:40:40.175407 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.175348    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7e4b380-d10c-4d5d-b399-7797eb58b804" containerName="istio-proxy"
Jul 24 00:40:40.177915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.177880    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:40.180330 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.180290    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bd6ad2bc-serving-cert\""
Jul 24 00:40:40.180427 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.180346    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nv7dj\""
Jul 24 00:40:40.180553 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.180531    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 00:40:40.180604 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.180553    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 00:40:40.188952 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.188932    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"]
Jul 24 00:40:40.323656 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.323619    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-maas-api-tls\") pod \"maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:40.323841 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.323670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbcv\" (UniqueName: \"kubernetes.io/projected/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-kube-api-access-wwbcv\") pod \"maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:40.323841 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.323779    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-openshift-service-ca\") pod \"maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:40.424431 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.424392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbcv\" (UniqueName: \"kubernetes.io/projected/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-kube-api-access-wwbcv\") pod \"maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:40.424578 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.424449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-openshift-service-ca\") pod \"maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:40.424578 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.424500    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-maas-api-tls\") pod \"maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:40.424652 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:40:40.424583    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-serving-cert: secret "maas-api-e2e-derive-bd6ad2bc-serving-cert" not found
Jul 24 00:40:40.424652 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:40:40.424637    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-maas-api-tls podName:ea667c5b-fbc3-4e17-96b9-2aae3f9d1794 nodeName:}" failed. No retries permitted until 2026-07-24 00:40:40.924620676 +0000 UTC m=+1803.772939394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-maas-api-tls") pod "maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq" (UID: "ea667c5b-fbc3-4e17-96b9-2aae3f9d1794") : secret "maas-api-e2e-derive-bd6ad2bc-serving-cert" not found
Jul 24 00:40:40.425144 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.425099    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-openshift-service-ca\") pod \"maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:40.432761 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.432734    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbcv\" (UniqueName: \"kubernetes.io/projected/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-kube-api-access-wwbcv\") pod \"maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:40.907564 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.907532    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"]
Jul 24 00:40:40.910743 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.910721    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:40:40.917062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.917038    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bd6ad2bc\""
Jul 24 00:40:40.917189 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.917036    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bd6ad2bc-dockercfg-7mzjg\""
Jul 24 00:40:40.929223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.929197    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-maas-api-tls\") pod \"maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:40.931397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.931372    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"]
Jul 24 00:40:40.932027 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:40.932004    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-maas-api-tls\") pod \"maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:41.029754 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.029714    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c09728-fcc4-45d5-8389-3d3070a591dd-plugins-config-volume\") pod \"payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r\" (UID: \"27c09728-fcc4-45d5-8389-3d3070a591dd\") " pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:40:41.029922 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.029781    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmwtm\" (UniqueName: \"kubernetes.io/projected/27c09728-fcc4-45d5-8389-3d3070a591dd-kube-api-access-rmwtm\") pod \"payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r\" (UID: \"27c09728-fcc4-45d5-8389-3d3070a591dd\") " pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:40:41.089036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.089009    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:41.131232 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.131201    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwtm\" (UniqueName: \"kubernetes.io/projected/27c09728-fcc4-45d5-8389-3d3070a591dd-kube-api-access-rmwtm\") pod \"payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r\" (UID: \"27c09728-fcc4-45d5-8389-3d3070a591dd\") " pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:40:41.131403 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.131290    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c09728-fcc4-45d5-8389-3d3070a591dd-plugins-config-volume\") pod \"payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r\" (UID: \"27c09728-fcc4-45d5-8389-3d3070a591dd\") " pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:40:41.132002 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.131982    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c09728-fcc4-45d5-8389-3d3070a591dd-plugins-config-volume\") pod \"payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r\" (UID: \"27c09728-fcc4-45d5-8389-3d3070a591dd\") " pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:40:41.140926 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.140897    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwtm\" (UniqueName: \"kubernetes.io/projected/27c09728-fcc4-45d5-8389-3d3070a591dd-kube-api-access-rmwtm\") pod \"payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r\" (UID: \"27c09728-fcc4-45d5-8389-3d3070a591dd\") " pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:40:41.219157 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.219121    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"]
Jul 24 00:40:41.221328 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.221293    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:40:41.221902 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:40:41.221876    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea667c5b_fbc3_4e17_96b9_2aae3f9d1794.slice/crio-ccefbb5f0e42818f7bae2bf7d92e843ac723f26c1e0a7062d55adee41c5e8bcf WatchSource:0}: Error finding container ccefbb5f0e42818f7bae2bf7d92e843ac723f26c1e0a7062d55adee41c5e8bcf: Status 404 returned error can't find the container with id ccefbb5f0e42818f7bae2bf7d92e843ac723f26c1e0a7062d55adee41c5e8bcf
Jul 24 00:40:41.350555 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.350528    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"]
Jul 24 00:40:41.352954 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:40:41.352928    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c09728_fcc4_45d5_8389_3d3070a591dd.slice/crio-c72e061c88c49479d5c7dcda5ac377264ea7e91ca89dc12e7ebb8a41566f7fea WatchSource:0}: Error finding container c72e061c88c49479d5c7dcda5ac377264ea7e91ca89dc12e7ebb8a41566f7fea: Status 404 returned error can't find the container with id c72e061c88c49479d5c7dcda5ac377264ea7e91ca89dc12e7ebb8a41566f7fea
Jul 24 00:40:41.745736 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.745698    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r" event={"ID":"27c09728-fcc4-45d5-8389-3d3070a591dd","Type":"ContainerStarted","Data":"92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8"}
Jul 24 00:40:41.745931 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.745763    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r" event={"ID":"27c09728-fcc4-45d5-8389-3d3070a591dd","Type":"ContainerStarted","Data":"c72e061c88c49479d5c7dcda5ac377264ea7e91ca89dc12e7ebb8a41566f7fea"}
Jul 24 00:40:41.745931 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.745794    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:40:41.751857 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.751827    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq" event={"ID":"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794","Type":"ContainerStarted","Data":"ccefbb5f0e42818f7bae2bf7d92e843ac723f26c1e0a7062d55adee41c5e8bcf"}
Jul 24 00:40:41.766378 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:41.766339    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r" podStartSLOduration=1.76632245 podStartE2EDuration="1.76632245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:40:41.765647202 +0000 UTC m=+1804.613965943" watchObservedRunningTime="2026-07-24 00:40:41.76632245 +0000 UTC m=+1804.614641191"
Jul 24 00:40:42.752537 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:42.752494    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq" event={"ID":"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794","Type":"ContainerStarted","Data":"1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e"}
Jul 24 00:40:42.752923 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:42.752797    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:42.770531 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:42.770481    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq" podStartSLOduration=2.045698935 podStartE2EDuration="2.770468356s" totalImagesPullingTime="724.769421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:40:42.769386079 +0000 UTC m=+1805.617704822" watchObservedRunningTime="2026-07-24 00:40:42.770468356 +0000 UTC m=+1805.618787098"
Jul 24 00:40:49.769153 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:49.769119    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:40:52.754722 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:52.754691    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:40:53.799078 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:53.799046    2574 generic.go:356] "Generic (PLEG): container finished" podID="99965c9c-338f-4cb0-8277-a261189678a1" containerID="cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d" exitCode=0
Jul 24 00:40:53.799488 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:53.799118    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" event={"ID":"99965c9c-338f-4cb0-8277-a261189678a1","Type":"ContainerDied","Data":"cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d"}
Jul 24 00:40:53.799586 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:53.799546    2574 scope.go:117] "RemoveContainer" containerID="cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d"
Jul 24 00:40:53.800579 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:53.800553    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:40:53.800647 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:53.800617    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:40:53.800701 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:53.800651    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:40:54.734938 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:54.734903    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:54.805262 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:54.805226    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" event={"ID":"99965c9c-338f-4cb0-8277-a261189678a1","Type":"ContainerStarted","Data":"0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b"}
Jul 24 00:40:55.688857 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:55.688823    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:55.689856 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:55.689835    2574 patch_prober.go:28] interesting pod/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 24 00:40:55.689918 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:55.689882    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" podUID="99965c9c-338f-4cb0-8277-a261189678a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 24 00:40:56.688289 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:56.688251    2574 patch_prober.go:28] interesting pod/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 24 00:40:56.688700 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:56.688332    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" podUID="99965c9c-338f-4cb0-8277-a261189678a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 24 00:40:57.689049 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:57.689007    2574 patch_prober.go:28] interesting pod/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 24 00:40:57.689464 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:57.689066    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" podUID="99965c9c-338f-4cb0-8277-a261189678a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 24 00:40:58.688558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:58.688516    2574 patch_prober.go:28] interesting pod/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 24 00:40:58.688759 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:58.688575    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" podUID="99965c9c-338f-4cb0-8277-a261189678a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 24 00:40:59.692747 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:59.692716    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:59.693205 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:59.693107    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:40:59.693742 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:40:59.693723    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:41:15.491616 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.491580    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"]
Jul 24 00:41:15.492121 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.491834    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq" podUID="ea667c5b-fbc3-4e17-96b9-2aae3f9d1794" containerName="maas-api" containerID="cri-o://1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e" gracePeriod=30
Jul 24 00:41:15.532934 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.532903    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"]
Jul 24 00:41:15.533126 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.533104    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r" podUID="27c09728-fcc4-45d5-8389-3d3070a591dd" containerName="payload-processing" containerID="cri-o://92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8" gracePeriod=30
Jul 24 00:41:15.774617 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.774591    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:41:15.804838 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.804810    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:41:15.826733 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.826705    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-maas-api-tls\") pod \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") "
Jul 24 00:41:15.826856 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.826754    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwbcv\" (UniqueName: \"kubernetes.io/projected/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-kube-api-access-wwbcv\") pod \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") "
Jul 24 00:41:15.826856 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.826841    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-openshift-service-ca\") pod \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\" (UID: \"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794\") "
Jul 24 00:41:15.827256 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.827228    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea667c5b-fbc3-4e17-96b9-2aae3f9d1794" (UID: "ea667c5b-fbc3-4e17-96b9-2aae3f9d1794"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:41:15.828818 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.828797    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea667c5b-fbc3-4e17-96b9-2aae3f9d1794" (UID: "ea667c5b-fbc3-4e17-96b9-2aae3f9d1794"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:41:15.828892 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.828853    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-kube-api-access-wwbcv" (OuterVolumeSpecName: "kube-api-access-wwbcv") pod "ea667c5b-fbc3-4e17-96b9-2aae3f9d1794" (UID: "ea667c5b-fbc3-4e17-96b9-2aae3f9d1794"). InnerVolumeSpecName "kube-api-access-wwbcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:41:15.890464 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.890431    2574 generic.go:356] "Generic (PLEG): container finished" podID="ea667c5b-fbc3-4e17-96b9-2aae3f9d1794" containerID="1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e" exitCode=0
Jul 24 00:41:15.890629 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.890481    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"
Jul 24 00:41:15.890629 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.890521    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq" event={"ID":"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794","Type":"ContainerDied","Data":"1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e"}
Jul 24 00:41:15.890629 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.890579    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq" event={"ID":"ea667c5b-fbc3-4e17-96b9-2aae3f9d1794","Type":"ContainerDied","Data":"ccefbb5f0e42818f7bae2bf7d92e843ac723f26c1e0a7062d55adee41c5e8bcf"}
Jul 24 00:41:15.890629 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.890610    2574 scope.go:117] "RemoveContainer" containerID="1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e"
Jul 24 00:41:15.891876 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.891854    2574 generic.go:356] "Generic (PLEG): container finished" podID="27c09728-fcc4-45d5-8389-3d3070a591dd" containerID="92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8" exitCode=0
Jul 24 00:41:15.891955 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.891885    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r" event={"ID":"27c09728-fcc4-45d5-8389-3d3070a591dd","Type":"ContainerDied","Data":"92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8"}
Jul 24 00:41:15.891955 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.891915    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"
Jul 24 00:41:15.891955 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.891932    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r" event={"ID":"27c09728-fcc4-45d5-8389-3d3070a591dd","Type":"ContainerDied","Data":"c72e061c88c49479d5c7dcda5ac377264ea7e91ca89dc12e7ebb8a41566f7fea"}
Jul 24 00:41:15.899107 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.899089    2574 scope.go:117] "RemoveContainer" containerID="1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e"
Jul 24 00:41:15.899362 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:41:15.899344    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e\": container with ID starting with 1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e not found: ID does not exist" containerID="1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e"
Jul 24 00:41:15.899466 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.899370    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e"} err="failed to get container status \"1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e\": rpc error: code = NotFound desc = could not find container \"1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e\": container with ID starting with 1ffbb9388593cc336ea8816c4c3e5254d36c300d23da2cff0aef7bede93c4b8e not found: ID does not exist"
Jul 24 00:41:15.899466 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.899386    2574 scope.go:117] "RemoveContainer" containerID="92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8"
Jul 24 00:41:15.906421 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.906402    2574 scope.go:117] "RemoveContainer" containerID="92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8"
Jul 24 00:41:15.906667 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:41:15.906645    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8\": container with ID starting with 92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8 not found: ID does not exist" containerID="92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8"
Jul 24 00:41:15.906727 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.906677    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8"} err="failed to get container status \"92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8\": rpc error: code = NotFound desc = could not find container \"92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8\": container with ID starting with 92e3b579ca9c099d86a5f6ebd2be563f7970f57f87d35b21d82dd87cea8fc3a8 not found: ID does not exist"
Jul 24 00:41:15.913540 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.913516    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"]
Jul 24 00:41:15.916085 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.916064    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd6ad2bc-66b6664658-w9fzq"]
Jul 24 00:41:15.927667 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.927644    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmwtm\" (UniqueName: \"kubernetes.io/projected/27c09728-fcc4-45d5-8389-3d3070a591dd-kube-api-access-rmwtm\") pod \"27c09728-fcc4-45d5-8389-3d3070a591dd\" (UID: \"27c09728-fcc4-45d5-8389-3d3070a591dd\") "
Jul 24 00:41:15.927756 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.927748    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c09728-fcc4-45d5-8389-3d3070a591dd-plugins-config-volume\") pod \"27c09728-fcc4-45d5-8389-3d3070a591dd\" (UID: \"27c09728-fcc4-45d5-8389-3d3070a591dd\") "
Jul 24 00:41:15.927887 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.927876    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:15.927925 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.927890    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwbcv\" (UniqueName: \"kubernetes.io/projected/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-kube-api-access-wwbcv\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:15.927925 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.927903    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:15.928079 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.928061    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c09728-fcc4-45d5-8389-3d3070a591dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27c09728-fcc4-45d5-8389-3d3070a591dd" (UID: "27c09728-fcc4-45d5-8389-3d3070a591dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:41:15.929518 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:15.929500    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c09728-fcc4-45d5-8389-3d3070a591dd-kube-api-access-rmwtm" (OuterVolumeSpecName: "kube-api-access-rmwtm") pod "27c09728-fcc4-45d5-8389-3d3070a591dd" (UID: "27c09728-fcc4-45d5-8389-3d3070a591dd"). InnerVolumeSpecName "kube-api-access-rmwtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:41:16.028532 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:16.028461    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c09728-fcc4-45d5-8389-3d3070a591dd-plugins-config-volume\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:16.028532 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:16.028484    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmwtm\" (UniqueName: \"kubernetes.io/projected/27c09728-fcc4-45d5-8389-3d3070a591dd-kube-api-access-rmwtm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:16.215354 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:16.215305    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"]
Jul 24 00:41:16.220015 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:16.219991    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd6ad2bc-6d5b4b868b-v6j2r"]
Jul 24 00:41:17.751447 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:17.751408    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c09728-fcc4-45d5-8389-3d3070a591dd" path="/var/lib/kubelet/pods/27c09728-fcc4-45d5-8389-3d3070a591dd/volumes"
Jul 24 00:41:17.751825 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:17.751774    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea667c5b-fbc3-4e17-96b9-2aae3f9d1794" path="/var/lib/kubelet/pods/ea667c5b-fbc3-4e17-96b9-2aae3f9d1794/volumes"
Jul 24 00:41:20.259257 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:20.259222    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"]
Jul 24 00:41:20.259730 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:20.259450    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" podUID="99965c9c-338f-4cb0-8277-a261189678a1" containerName="istio-proxy" containerID="cri-o://0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b" gracePeriod=30
Jul 24 00:41:25.511141 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.511113    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:41:25.616007 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.615976    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99965c9c-338f-4cb0-8277-a261189678a1-istiod-ca-cert\") pod \"99965c9c-338f-4cb0-8277-a261189678a1\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") "
Jul 24 00:41:25.616175 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616012    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99965c9c-338f-4cb0-8277-a261189678a1-istio-podinfo\") pod \"99965c9c-338f-4cb0-8277-a261189678a1\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") "
Jul 24 00:41:25.616175 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616038    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-credential-socket\") pod \"99965c9c-338f-4cb0-8277-a261189678a1\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") "
Jul 24 00:41:25.616175 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616095    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-istio-token\") pod \"99965c9c-338f-4cb0-8277-a261189678a1\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") "
Jul 24 00:41:25.616175 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616115    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qttzk\" (UniqueName: \"kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-kube-api-access-qttzk\") pod \"99965c9c-338f-4cb0-8277-a261189678a1\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") "
Jul 24 00:41:25.616175 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616138    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-socket\") pod \"99965c9c-338f-4cb0-8277-a261189678a1\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") "
Jul 24 00:41:25.616464 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616175    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-envoy\") pod \"99965c9c-338f-4cb0-8277-a261189678a1\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") "
Jul 24 00:41:25.616464 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616203    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-data\") pod \"99965c9c-338f-4cb0-8277-a261189678a1\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") "
Jul 24 00:41:25.616464 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616239    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-certs\") pod \"99965c9c-338f-4cb0-8277-a261189678a1\" (UID: \"99965c9c-338f-4cb0-8277-a261189678a1\") "
Jul 24 00:41:25.616464 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616370    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99965c9c-338f-4cb0-8277-a261189678a1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99965c9c-338f-4cb0-8277-a261189678a1" (UID: "99965c9c-338f-4cb0-8277-a261189678a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:41:25.616464 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616447    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99965c9c-338f-4cb0-8277-a261189678a1" (UID: "99965c9c-338f-4cb0-8277-a261189678a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:41:25.616720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616512    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99965c9c-338f-4cb0-8277-a261189678a1-istiod-ca-cert\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:25.616720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616533    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-credential-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:25.616720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616596    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99965c9c-338f-4cb0-8277-a261189678a1" (UID: "99965c9c-338f-4cb0-8277-a261189678a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:41:25.616877 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.616606    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99965c9c-338f-4cb0-8277-a261189678a1" (UID: "99965c9c-338f-4cb0-8277-a261189678a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:41:25.617129 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.617110    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-data" (OuterVolumeSpecName: "istio-data") pod "99965c9c-338f-4cb0-8277-a261189678a1" (UID: "99965c9c-338f-4cb0-8277-a261189678a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:41:25.618304 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.618273    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99965c9c-338f-4cb0-8277-a261189678a1" (UID: "99965c9c-338f-4cb0-8277-a261189678a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:41:25.618304 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.618291    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99965c9c-338f-4cb0-8277-a261189678a1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99965c9c-338f-4cb0-8277-a261189678a1" (UID: "99965c9c-338f-4cb0-8277-a261189678a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 00:41:25.618824 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.618789    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-istio-token" (OuterVolumeSpecName: "istio-token") pod "99965c9c-338f-4cb0-8277-a261189678a1" (UID: "99965c9c-338f-4cb0-8277-a261189678a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:41:25.618824 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.618789    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-kube-api-access-qttzk" (OuterVolumeSpecName: "kube-api-access-qttzk") pod "99965c9c-338f-4cb0-8277-a261189678a1" (UID: "99965c9c-338f-4cb0-8277-a261189678a1"). InnerVolumeSpecName "kube-api-access-qttzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:41:25.717935 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.717905    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-istio-token\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:25.717935 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.717929    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qttzk\" (UniqueName: \"kubernetes.io/projected/99965c9c-338f-4cb0-8277-a261189678a1-kube-api-access-qttzk\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:25.717935 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.717940    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:25.718134 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.717949    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-envoy\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:25.718134 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.717957    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-istio-data\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:25.718134 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.717964    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99965c9c-338f-4cb0-8277-a261189678a1-workload-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:25.718134 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.717972    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99965c9c-338f-4cb0-8277-a261189678a1-istio-podinfo\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:41:25.932902 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.932818    2574 generic.go:356] "Generic (PLEG): container finished" podID="99965c9c-338f-4cb0-8277-a261189678a1" containerID="0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b" exitCode=0
Jul 24 00:41:25.932902 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.932860    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"
Jul 24 00:41:25.933087 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.932857    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" event={"ID":"99965c9c-338f-4cb0-8277-a261189678a1","Type":"ContainerDied","Data":"0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b"}
Jul 24 00:41:25.933087 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.932973    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x" event={"ID":"99965c9c-338f-4cb0-8277-a261189678a1","Type":"ContainerDied","Data":"1a0059ff3f22659f7e33649509b32febfce2b4f04e91b7b736265006531dbd57"}
Jul 24 00:41:25.933087 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.933003    2574 scope.go:117] "RemoveContainer" containerID="0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b"
Jul 24 00:41:25.941270 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.941247    2574 scope.go:117] "RemoveContainer" containerID="cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d"
Jul 24 00:41:25.948336 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.948301    2574 scope.go:117] "RemoveContainer" containerID="0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b"
Jul 24 00:41:25.948875 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:41:25.948690    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b\": container with ID starting with 0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b not found: ID does not exist" containerID="0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b"
Jul 24 00:41:25.948875 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.948764    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b"} err="failed to get container status \"0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b\": rpc error: code = NotFound desc = could not find container \"0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b\": container with ID starting with 0f746572115ac089651ca0357eb14ea9b405ac086689c33ad6c40ffdfb0f220b not found: ID does not exist"
Jul 24 00:41:25.948875 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.948797    2574 scope.go:117] "RemoveContainer" containerID="cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d"
Jul 24 00:41:25.949142 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:41:25.949106    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d\": container with ID starting with cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d not found: ID does not exist" containerID="cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d"
Jul 24 00:41:25.949142 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.949134    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d"} err="failed to get container status \"cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d\": rpc error: code = NotFound desc = could not find container \"cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d\": container with ID starting with cd374ed2e79a00353b0fd859a689836da6157d81e33d3aa4d3ee68b9e724661d not found: ID does not exist"
Jul 24 00:41:25.950374 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.950352    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"]
Jul 24 00:41:25.952050 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:25.952032    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-bd6ad2bc-openshift-default-778bf9554d-cbs7x"]
Jul 24 00:41:27.751457 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:41:27.751422    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99965c9c-338f-4cb0-8277-a261189678a1" path="/var/lib/kubelet/pods/99965c9c-338f-4cb0-8277-a261189678a1/volumes"
Jul 24 00:42:57.752590 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752558    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"]
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752814    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c09728-fcc4-45d5-8389-3d3070a591dd" containerName="payload-processing"
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752823    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c09728-fcc4-45d5-8389-3d3070a591dd" containerName="payload-processing"
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752836    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea667c5b-fbc3-4e17-96b9-2aae3f9d1794" containerName="maas-api"
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752842    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea667c5b-fbc3-4e17-96b9-2aae3f9d1794" containerName="maas-api"
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752851    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99965c9c-338f-4cb0-8277-a261189678a1" containerName="istio-proxy"
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752856    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="99965c9c-338f-4cb0-8277-a261189678a1" containerName="istio-proxy"
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752865    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99965c9c-338f-4cb0-8277-a261189678a1" containerName="istio-proxy"
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752870    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="99965c9c-338f-4cb0-8277-a261189678a1" containerName="istio-proxy"
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752910    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="99965c9c-338f-4cb0-8277-a261189678a1" containerName="istio-proxy"
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752917    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c09728-fcc4-45d5-8389-3d3070a591dd" containerName="payload-processing"
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752924    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea667c5b-fbc3-4e17-96b9-2aae3f9d1794" containerName="maas-api"
Jul 24 00:42:57.753062 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.752930    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="99965c9c-338f-4cb0-8277-a261189678a1" containerName="istio-proxy"
Jul 24 00:42:57.755543 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.755526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.758094 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.758069    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7f92c6ce-openshift-default-dockercfg-5sq9n\""
Jul 24 00:42:57.771639 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.771617    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"]
Jul 24 00:42:57.893495 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.893455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-podinfo\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.893495 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.893493    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-token\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.893761 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.893591    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2288b1-50ff-444a-9313-49bd2218a3b0-istiod-ca-cert\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.893761 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.893641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-data\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.893761 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.893680    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-socket\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.893761 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.893703    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-envoy\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.893989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.893885    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-certs\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.893989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.893920    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-credential-socket\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.893989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.893966    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5qxf\" (UniqueName: \"kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-kube-api-access-c5qxf\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.994571 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.994538    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2288b1-50ff-444a-9313-49bd2218a3b0-istiod-ca-cert\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.994571 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.994574    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-data\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.994840 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.994605    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-socket\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.994840 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.994654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-envoy\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.994840 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.994693    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-certs\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.994840 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.994710    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-credential-socket\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.994840 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.994734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5qxf\" (UniqueName: \"kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-kube-api-access-c5qxf\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.994840 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.994800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-podinfo\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.994840 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.994828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-token\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.995222 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.994991    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-data\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.995222 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.995072    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-credential-socket\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.995341 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.995277    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-socket\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.995439 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.995417    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-certs\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.995797 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.995766    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2288b1-50ff-444a-9313-49bd2218a3b0-istiod-ca-cert\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.996991 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.996970    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-envoy\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:57.997294 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:57.997276    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-podinfo\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:58.005643 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:58.005580    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-token\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:58.005807 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:58.005790    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5qxf\" (UniqueName: \"kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-kube-api-access-c5qxf\") pod \"e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") " pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:58.065563 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:58.065531    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:58.211372 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:58.211344    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"]
Jul 24 00:42:58.214064 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:42:58.214037    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2288b1_50ff_444a_9313_49bd2218a3b0.slice/crio-876ee24af6bdbeb37e3e87cc090b2b0c7b9f67bec7c0510658ada98a572af5a5 WatchSource:0}: Error finding container 876ee24af6bdbeb37e3e87cc090b2b0c7b9f67bec7c0510658ada98a572af5a5: Status 404 returned error can't find the container with id 876ee24af6bdbeb37e3e87cc090b2b0c7b9f67bec7c0510658ada98a572af5a5
Jul 24 00:42:58.216795 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:58.216758    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:42:58.216880 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:58.216847    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:42:58.216920 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:58.216909    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:42:58.298458 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:58.298429    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" event={"ID":"7d2288b1-50ff-444a-9313-49bd2218a3b0","Type":"ContainerStarted","Data":"4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a"}
Jul 24 00:42:58.298599 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:58.298487    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" event={"ID":"7d2288b1-50ff-444a-9313-49bd2218a3b0","Type":"ContainerStarted","Data":"876ee24af6bdbeb37e3e87cc090b2b0c7b9f67bec7c0510658ada98a572af5a5"}
Jul 24 00:42:58.318797 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:58.318730    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" podStartSLOduration=1.318714692 podStartE2EDuration="1.318714692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:42:58.317839731 +0000 UTC m=+1941.166158475" watchObservedRunningTime="2026-07-24 00:42:58.318714692 +0000 UTC m=+1941.167033433"
Jul 24 00:42:59.066472 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:59.066435    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:59.071890 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:59.071861    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:59.302542 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:59.302510    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:42:59.303534 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:42:59.303514    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:43:00.849919 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:00.849883    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"]
Jul 24 00:43:00.853580 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:00.853559    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:00.856423 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:00.856387    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7f92c6ce-serving-cert\""
Jul 24 00:43:00.856586 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:00.856401    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 00:43:00.856808 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:00.856793    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nv7dj\""
Jul 24 00:43:00.857586 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:00.857563    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 00:43:00.865339 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:00.865296    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"]
Jul 24 00:43:00.922677 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:00.922638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b2b888-a229-45b0-b02c-7716f7926194-openshift-service-ca\") pod \"maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:00.922875 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:00.922692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07b2b888-a229-45b0-b02c-7716f7926194-maas-api-tls\") pod \"maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:00.922875 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:00.922788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkq7g\" (UniqueName: \"kubernetes.io/projected/07b2b888-a229-45b0-b02c-7716f7926194-kube-api-access-bkq7g\") pod \"maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:01.023930 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.023895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b2b888-a229-45b0-b02c-7716f7926194-openshift-service-ca\") pod \"maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:01.024131 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.023947    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07b2b888-a229-45b0-b02c-7716f7926194-maas-api-tls\") pod \"maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:01.024131 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.024016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkq7g\" (UniqueName: \"kubernetes.io/projected/07b2b888-a229-45b0-b02c-7716f7926194-kube-api-access-bkq7g\") pod \"maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:01.024267 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:43:01.024151    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-serving-cert: secret "maas-api-e2e-mt-7f92c6ce-serving-cert" not found
Jul 24 00:43:01.024267 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:43:01.024239    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07b2b888-a229-45b0-b02c-7716f7926194-maas-api-tls podName:07b2b888-a229-45b0-b02c-7716f7926194 nodeName:}" failed. No retries permitted until 2026-07-24 00:43:01.524214743 +0000 UTC m=+1944.372533478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07b2b888-a229-45b0-b02c-7716f7926194-maas-api-tls") pod "maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6" (UID: "07b2b888-a229-45b0-b02c-7716f7926194") : secret "maas-api-e2e-mt-7f92c6ce-serving-cert" not found
Jul 24 00:43:01.024731 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.024707    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b2b888-a229-45b0-b02c-7716f7926194-openshift-service-ca\") pod \"maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:01.033006 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.032977    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkq7g\" (UniqueName: \"kubernetes.io/projected/07b2b888-a229-45b0-b02c-7716f7926194-kube-api-access-bkq7g\") pod \"maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:01.528379 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.528338    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07b2b888-a229-45b0-b02c-7716f7926194-maas-api-tls\") pod \"maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:01.530722 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.530695    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07b2b888-a229-45b0-b02c-7716f7926194-maas-api-tls\") pod \"maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:01.595623 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.595590    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"]
Jul 24 00:43:01.598738 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.598715    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:01.602635 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.602611    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7f92c6ce\""
Jul 24 00:43:01.602764 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.602690    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7f92c6ce-dockercfg-r92sf\""
Jul 24 00:43:01.616592 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.616562    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"]
Jul 24 00:43:01.729761 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.729724    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmx4n\" (UniqueName: \"kubernetes.io/projected/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-kube-api-access-tmx4n\") pod \"payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r\" (UID: \"50362eaf-93ee-4f12-b4e2-8ea3b6edb164\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:01.729917 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.729770    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r\" (UID: \"50362eaf-93ee-4f12-b4e2-8ea3b6edb164\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:01.768810 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.768772    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:01.833959 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.832545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmx4n\" (UniqueName: \"kubernetes.io/projected/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-kube-api-access-tmx4n\") pod \"payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r\" (UID: \"50362eaf-93ee-4f12-b4e2-8ea3b6edb164\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:01.833959 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.832606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r\" (UID: \"50362eaf-93ee-4f12-b4e2-8ea3b6edb164\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:01.833959 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.833503    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r\" (UID: \"50362eaf-93ee-4f12-b4e2-8ea3b6edb164\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:01.846343 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.843681    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmx4n\" (UniqueName: \"kubernetes.io/projected/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-kube-api-access-tmx4n\") pod \"payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r\" (UID: \"50362eaf-93ee-4f12-b4e2-8ea3b6edb164\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:01.909460 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:01.909427    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:02.048064 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:02.047970    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"]
Jul 24 00:43:02.050839 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:43:02.050805    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50362eaf_93ee_4f12_b4e2_8ea3b6edb164.slice/crio-6f22375135e410fda4a9a0033c8160be30e68da73b2e730d9af799fbf684a5bf WatchSource:0}: Error finding container 6f22375135e410fda4a9a0033c8160be30e68da73b2e730d9af799fbf684a5bf: Status 404 returned error can't find the container with id 6f22375135e410fda4a9a0033c8160be30e68da73b2e730d9af799fbf684a5bf
Jul 24 00:43:02.130788 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:02.130755    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"]
Jul 24 00:43:02.133472 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:43:02.133446    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b2b888_a229_45b0_b02c_7716f7926194.slice/crio-b093ee5c04040dc42e401eb38ba704e4374e7d0338f474f0d414cf4ed7e175c6 WatchSource:0}: Error finding container b093ee5c04040dc42e401eb38ba704e4374e7d0338f474f0d414cf4ed7e175c6: Status 404 returned error can't find the container with id b093ee5c04040dc42e401eb38ba704e4374e7d0338f474f0d414cf4ed7e175c6
Jul 24 00:43:02.317022 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:02.316980    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r" event={"ID":"50362eaf-93ee-4f12-b4e2-8ea3b6edb164","Type":"ContainerStarted","Data":"d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa"}
Jul 24 00:43:02.317205 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:02.317054    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:02.317205 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:02.317078    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r" event={"ID":"50362eaf-93ee-4f12-b4e2-8ea3b6edb164","Type":"ContainerStarted","Data":"6f22375135e410fda4a9a0033c8160be30e68da73b2e730d9af799fbf684a5bf"}
Jul 24 00:43:02.318272 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:02.318247    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6" event={"ID":"07b2b888-a229-45b0-b02c-7716f7926194","Type":"ContainerStarted","Data":"b093ee5c04040dc42e401eb38ba704e4374e7d0338f474f0d414cf4ed7e175c6"}
Jul 24 00:43:02.338385 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:02.338295    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r" podStartSLOduration=1.338273193 podStartE2EDuration="1.338273193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:43:02.337593906 +0000 UTC m=+1945.185912649" watchObservedRunningTime="2026-07-24 00:43:02.338273193 +0000 UTC m=+1945.186591935"
Jul 24 00:43:03.326992 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:03.326950    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6" event={"ID":"07b2b888-a229-45b0-b02c-7716f7926194","Type":"ContainerStarted","Data":"1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879"}
Jul 24 00:43:03.327594 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:03.327039    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:03.344673 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:03.344592    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6" podStartSLOduration=2.789356196 podStartE2EDuration="3.34457405s" totalImagesPullingTime="555.217854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:43:03.343849902 +0000 UTC m=+1946.192168646" watchObservedRunningTime="2026-07-24 00:43:03.34457405 +0000 UTC m=+1946.192892792"
Jul 24 00:43:10.339685 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:10.339651    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:13.328502 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:13.328465    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:14.304437 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:14.304400    2574 patch_prober.go:28] interesting pod/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 24 00:43:14.304646 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:14.304459    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 24 00:43:14.381970 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:14.381938    2574 generic.go:356] "Generic (PLEG): container finished" podID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerID="4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a" exitCode=0
Jul 24 00:43:14.382390 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:14.382012    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" event={"ID":"7d2288b1-50ff-444a-9313-49bd2218a3b0","Type":"ContainerDied","Data":"4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a"}
Jul 24 00:43:14.382456 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:14.382406    2574 scope.go:117] "RemoveContainer" containerID="4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a"
Jul 24 00:43:14.383383 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:14.383355    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:43:14.383466 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:14.383423    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:43:14.383466 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:14.383453    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:43:15.387929 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:15.387894    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" event={"ID":"7d2288b1-50ff-444a-9313-49bd2218a3b0","Type":"ContainerStarted","Data":"38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0"}
Jul 24 00:43:16.066366 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:16.066330    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:43:16.066552 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:16.066488    2574 patch_prober.go:28] interesting pod/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 24 00:43:16.066602 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:16.066550    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 24 00:43:17.066688 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:17.066651    2574 patch_prober.go:28] interesting pod/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 24 00:43:17.067098 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:17.066713    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 24 00:43:18.066925 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:18.066884    2574 patch_prober.go:28] interesting pod/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 24 00:43:18.067336 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:18.066945    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 24 00:43:19.066701 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:19.066663    2574 patch_prober.go:28] interesting pod/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 24 00:43:19.066877 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:19.066722    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 24 00:43:20.070468 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:20.070433    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:43:20.070893 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:20.070705    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:43:20.071396 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:20.071379    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:43:23.044340 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.041669    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"]
Jul 24 00:43:23.044340 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.041983    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6" podUID="07b2b888-a229-45b0-b02c-7716f7926194" containerName="maas-api" containerID="cri-o://1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879" gracePeriod=30
Jul 24 00:43:23.107467 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.107360    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"]
Jul 24 00:43:23.107621 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.107595    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r" podUID="50362eaf-93ee-4f12-b4e2-8ea3b6edb164" containerName="payload-pre-processing" containerID="cri-o://d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa" gracePeriod=30
Jul 24 00:43:23.342970 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.342944    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:23.375143 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.375120    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:23.423739 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.423710    2574 generic.go:356] "Generic (PLEG): container finished" podID="50362eaf-93ee-4f12-b4e2-8ea3b6edb164" containerID="d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa" exitCode=0
Jul 24 00:43:23.423902 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.423769    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"
Jul 24 00:43:23.423902 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.423793    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r" event={"ID":"50362eaf-93ee-4f12-b4e2-8ea3b6edb164","Type":"ContainerDied","Data":"d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa"}
Jul 24 00:43:23.423902 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.423848    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r" event={"ID":"50362eaf-93ee-4f12-b4e2-8ea3b6edb164","Type":"ContainerDied","Data":"6f22375135e410fda4a9a0033c8160be30e68da73b2e730d9af799fbf684a5bf"}
Jul 24 00:43:23.423902 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.423879    2574 scope.go:117] "RemoveContainer" containerID="d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa"
Jul 24 00:43:23.425121 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.425100    2574 generic.go:356] "Generic (PLEG): container finished" podID="07b2b888-a229-45b0-b02c-7716f7926194" containerID="1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879" exitCode=0
Jul 24 00:43:23.425209 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.425136    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"
Jul 24 00:43:23.425209 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.425185    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6" event={"ID":"07b2b888-a229-45b0-b02c-7716f7926194","Type":"ContainerDied","Data":"1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879"}
Jul 24 00:43:23.425275 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.425233    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6" event={"ID":"07b2b888-a229-45b0-b02c-7716f7926194","Type":"ContainerDied","Data":"b093ee5c04040dc42e401eb38ba704e4374e7d0338f474f0d414cf4ed7e175c6"}
Jul 24 00:43:23.427164 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.427148    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b2b888-a229-45b0-b02c-7716f7926194-openshift-service-ca\") pod \"07b2b888-a229-45b0-b02c-7716f7926194\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") "
Jul 24 00:43:23.427230 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.427172    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07b2b888-a229-45b0-b02c-7716f7926194-maas-api-tls\") pod \"07b2b888-a229-45b0-b02c-7716f7926194\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") "
Jul 24 00:43:23.427230 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.427197    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmx4n\" (UniqueName: \"kubernetes.io/projected/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-kube-api-access-tmx4n\") pod \"50362eaf-93ee-4f12-b4e2-8ea3b6edb164\" (UID: \"50362eaf-93ee-4f12-b4e2-8ea3b6edb164\") "
Jul 24 00:43:23.427230 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.427227    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-plugins-config-volume\") pod \"50362eaf-93ee-4f12-b4e2-8ea3b6edb164\" (UID: \"50362eaf-93ee-4f12-b4e2-8ea3b6edb164\") "
Jul 24 00:43:23.427415 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.427303    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkq7g\" (UniqueName: \"kubernetes.io/projected/07b2b888-a229-45b0-b02c-7716f7926194-kube-api-access-bkq7g\") pod \"07b2b888-a229-45b0-b02c-7716f7926194\" (UID: \"07b2b888-a229-45b0-b02c-7716f7926194\") "
Jul 24 00:43:23.427554 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.427529    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b2b888-a229-45b0-b02c-7716f7926194-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07b2b888-a229-45b0-b02c-7716f7926194" (UID: "07b2b888-a229-45b0-b02c-7716f7926194"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:43:23.427835 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.427751    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50362eaf-93ee-4f12-b4e2-8ea3b6edb164" (UID: "50362eaf-93ee-4f12-b4e2-8ea3b6edb164"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:43:23.429421 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.429388    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b2b888-a229-45b0-b02c-7716f7926194-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07b2b888-a229-45b0-b02c-7716f7926194" (UID: "07b2b888-a229-45b0-b02c-7716f7926194"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:43:23.429521 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.429457    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-kube-api-access-tmx4n" (OuterVolumeSpecName: "kube-api-access-tmx4n") pod "50362eaf-93ee-4f12-b4e2-8ea3b6edb164" (UID: "50362eaf-93ee-4f12-b4e2-8ea3b6edb164"). InnerVolumeSpecName "kube-api-access-tmx4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:43:23.429644 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.429628    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b2b888-a229-45b0-b02c-7716f7926194-kube-api-access-bkq7g" (OuterVolumeSpecName: "kube-api-access-bkq7g") pod "07b2b888-a229-45b0-b02c-7716f7926194" (UID: "07b2b888-a229-45b0-b02c-7716f7926194"). InnerVolumeSpecName "kube-api-access-bkq7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:43:23.432556 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.432539    2574 scope.go:117] "RemoveContainer" containerID="d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa"
Jul 24 00:43:23.432797 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:43:23.432775    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa\": container with ID starting with d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa not found: ID does not exist" containerID="d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa"
Jul 24 00:43:23.432843 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.432806    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa"} err="failed to get container status \"d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa\": rpc error: code = NotFound desc = could not find container \"d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa\": container with ID starting with d76bbb99ca6cf999f53ecb26bebacd122b4d26487c122b138c91693cc8433caa not found: ID does not exist"
Jul 24 00:43:23.432843 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.432823    2574 scope.go:117] "RemoveContainer" containerID="1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879"
Jul 24 00:43:23.439732 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.439717    2574 scope.go:117] "RemoveContainer" containerID="1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879"
Jul 24 00:43:23.439993 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:43:23.439977    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879\": container with ID starting with 1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879 not found: ID does not exist" containerID="1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879"
Jul 24 00:43:23.440059 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.439996    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879"} err="failed to get container status \"1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879\": rpc error: code = NotFound desc = could not find container \"1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879\": container with ID starting with 1145128edfae7188c86338238c01c798f57fa3a5c1228814f2f3de4e80498879 not found: ID does not exist"
Jul 24 00:43:23.528129 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.528098    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkq7g\" (UniqueName: \"kubernetes.io/projected/07b2b888-a229-45b0-b02c-7716f7926194-kube-api-access-bkq7g\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:23.528129 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.528124    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b2b888-a229-45b0-b02c-7716f7926194-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:23.528129 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.528135    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07b2b888-a229-45b0-b02c-7716f7926194-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:23.528378 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.528143    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmx4n\" (UniqueName: \"kubernetes.io/projected/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-kube-api-access-tmx4n\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:23.528378 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.528151    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50362eaf-93ee-4f12-b4e2-8ea3b6edb164-plugins-config-volume\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:23.752056 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.752022    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"]
Jul 24 00:43:23.752056 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.752058    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r"]
Jul 24 00:43:23.760881 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.760859    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"]
Jul 24 00:43:23.764872 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:23.764849    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f92c6ce-6f74b9599c-kx2b6"]
Jul 24 00:43:24.327704 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:24.327660    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f92c6ce-757679fbd8-z9v6r" podUID="50362eaf-93ee-4f12-b4e2-8ea3b6edb164" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.44:9004: i/o timeout"
Jul 24 00:43:25.751453 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:25.751420    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07b2b888-a229-45b0-b02c-7716f7926194" path="/var/lib/kubelet/pods/07b2b888-a229-45b0-b02c-7716f7926194/volumes"
Jul 24 00:43:25.751824 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:25.751758    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50362eaf-93ee-4f12-b4e2-8ea3b6edb164" path="/var/lib/kubelet/pods/50362eaf-93ee-4f12-b4e2-8ea3b6edb164/volumes"
Jul 24 00:43:33.987036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:33.986938    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"]
Jul 24 00:43:33.987512 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:33.987208    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy" containerID="cri-o://38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0" gracePeriod=30
Jul 24 00:43:35.073640 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:35.073598    2574 patch_prober.go:28] interesting pod/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 00:43:35.074090 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:35.073663    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 00:43:39.237110 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.237082    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:43:39.360949 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.360905    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2288b1-50ff-444a-9313-49bd2218a3b0-istiod-ca-cert\") pod \"7d2288b1-50ff-444a-9313-49bd2218a3b0\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") "
Jul 24 00:43:39.360949 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.360951    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-podinfo\") pod \"7d2288b1-50ff-444a-9313-49bd2218a3b0\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") "
Jul 24 00:43:39.361223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361001    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-token\") pod \"7d2288b1-50ff-444a-9313-49bd2218a3b0\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") "
Jul 24 00:43:39.361223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361018    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-credential-socket\") pod \"7d2288b1-50ff-444a-9313-49bd2218a3b0\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") "
Jul 24 00:43:39.361223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361048    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-envoy\") pod \"7d2288b1-50ff-444a-9313-49bd2218a3b0\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") "
Jul 24 00:43:39.361223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361066    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5qxf\" (UniqueName: \"kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-kube-api-access-c5qxf\") pod \"7d2288b1-50ff-444a-9313-49bd2218a3b0\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") "
Jul 24 00:43:39.361223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361127    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-certs\") pod \"7d2288b1-50ff-444a-9313-49bd2218a3b0\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") "
Jul 24 00:43:39.361223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361171    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-socket\") pod \"7d2288b1-50ff-444a-9313-49bd2218a3b0\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") "
Jul 24 00:43:39.361223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361203    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-data\") pod \"7d2288b1-50ff-444a-9313-49bd2218a3b0\" (UID: \"7d2288b1-50ff-444a-9313-49bd2218a3b0\") "
Jul 24 00:43:39.361562 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361294    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2288b1-50ff-444a-9313-49bd2218a3b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d2288b1-50ff-444a-9313-49bd2218a3b0" (UID: "7d2288b1-50ff-444a-9313-49bd2218a3b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:43:39.361921 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361877    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d2288b1-50ff-444a-9313-49bd2218a3b0" (UID: "7d2288b1-50ff-444a-9313-49bd2218a3b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:43:39.361921 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361906    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d2288b1-50ff-444a-9313-49bd2218a3b0" (UID: "7d2288b1-50ff-444a-9313-49bd2218a3b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:43:39.362109 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361948    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2288b1-50ff-444a-9313-49bd2218a3b0-istiod-ca-cert\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:39.363855 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.361571    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d2288b1-50ff-444a-9313-49bd2218a3b0" (UID: "7d2288b1-50ff-444a-9313-49bd2218a3b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:43:39.364164 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.364134    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d2288b1-50ff-444a-9313-49bd2218a3b0" (UID: "7d2288b1-50ff-444a-9313-49bd2218a3b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:43:39.365170 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.364911    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-kube-api-access-c5qxf" (OuterVolumeSpecName: "kube-api-access-c5qxf") pod "7d2288b1-50ff-444a-9313-49bd2218a3b0" (UID: "7d2288b1-50ff-444a-9313-49bd2218a3b0"). InnerVolumeSpecName "kube-api-access-c5qxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:43:39.365170 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.364939    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d2288b1-50ff-444a-9313-49bd2218a3b0" (UID: "7d2288b1-50ff-444a-9313-49bd2218a3b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 00:43:39.371852 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.371585    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d2288b1-50ff-444a-9313-49bd2218a3b0" (UID: "7d2288b1-50ff-444a-9313-49bd2218a3b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:43:39.371852 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.371648    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d2288b1-50ff-444a-9313-49bd2218a3b0" (UID: "7d2288b1-50ff-444a-9313-49bd2218a3b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:43:39.462731 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.462686    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-token\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:39.462731 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.462723    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-credential-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:39.462731 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.462733    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-envoy\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:39.462731 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.462744    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5qxf\" (UniqueName: \"kubernetes.io/projected/7d2288b1-50ff-444a-9313-49bd2218a3b0-kube-api-access-c5qxf\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:39.463009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.462754    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:39.463009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.462762    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-workload-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:39.463009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.462770    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-data\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:39.463009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.462777    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2288b1-50ff-444a-9313-49bd2218a3b0-istio-podinfo\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:43:39.494096 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.494066    2574 generic.go:356] "Generic (PLEG): container finished" podID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerID="38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0" exitCode=0
Jul 24 00:43:39.494219 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.494136    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"
Jul 24 00:43:39.494258 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.494141    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" event={"ID":"7d2288b1-50ff-444a-9313-49bd2218a3b0","Type":"ContainerDied","Data":"38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0"}
Jul 24 00:43:39.494293 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.494280    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz" event={"ID":"7d2288b1-50ff-444a-9313-49bd2218a3b0","Type":"ContainerDied","Data":"876ee24af6bdbeb37e3e87cc090b2b0c7b9f67bec7c0510658ada98a572af5a5"}
Jul 24 00:43:39.494347 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.494302    2574 scope.go:117] "RemoveContainer" containerID="38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0"
Jul 24 00:43:39.503574 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.503557    2574 scope.go:117] "RemoveContainer" containerID="4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a"
Jul 24 00:43:39.510849 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.510834    2574 scope.go:117] "RemoveContainer" containerID="38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0"
Jul 24 00:43:39.511121 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:43:39.511101    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0\": container with ID starting with 38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0 not found: ID does not exist" containerID="38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0"
Jul 24 00:43:39.511179 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.511130    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0"} err="failed to get container status \"38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0\": rpc error: code = NotFound desc = could not find container \"38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0\": container with ID starting with 38bbb88b72bbdb2fa41290e7c99e3497a12e6cbb3f257040dc379eb2145d59b0 not found: ID does not exist"
Jul 24 00:43:39.511179 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.511148    2574 scope.go:117] "RemoveContainer" containerID="4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a"
Jul 24 00:43:39.511378 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:43:39.511363    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a\": container with ID starting with 4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a not found: ID does not exist" containerID="4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a"
Jul 24 00:43:39.511425 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.511384    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a"} err="failed to get container status \"4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a\": rpc error: code = NotFound desc = could not find container \"4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a\": container with ID starting with 4809bfb907a242fc37fee014bcbcf8a53123114f9fbd66d21c020792fbd4532a not found: ID does not exist"
Jul 24 00:43:39.515713 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.515692    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"]
Jul 24 00:43:39.518727 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.518707    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7f92c6ce-openshift-default-6dd9b988cb-rtcdz"]
Jul 24 00:43:39.751789 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:39.751705    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" path="/var/lib/kubelet/pods/7d2288b1-50ff-444a-9313-49bd2218a3b0/volumes"
Jul 24 00:43:53.583586 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.583556    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"]
Jul 24 00:43:53.583949 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.583831    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b2b888-a229-45b0-b02c-7716f7926194" containerName="maas-api"
Jul 24 00:43:53.583949 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.583841    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b2b888-a229-45b0-b02c-7716f7926194" containerName="maas-api"
Jul 24 00:43:53.583949 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.583853    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50362eaf-93ee-4f12-b4e2-8ea3b6edb164" containerName="payload-pre-processing"
Jul 24 00:43:53.583949 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.583859    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="50362eaf-93ee-4f12-b4e2-8ea3b6edb164" containerName="payload-pre-processing"
Jul 24 00:43:53.583949 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.583873    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy"
Jul 24 00:43:53.583949 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.583880    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy"
Jul 24 00:43:53.584137 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.583969    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="07b2b888-a229-45b0-b02c-7716f7926194" containerName="maas-api"
Jul 24 00:43:53.584137 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.583984    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy"
Jul 24 00:43:53.584137 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.583993    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="50362eaf-93ee-4f12-b4e2-8ea3b6edb164" containerName="payload-pre-processing"
Jul 24 00:43:53.586012 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.585995    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.589257 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.589237    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-464c3345-openshift-default-dockercfg-pskxt\""
Jul 24 00:43:53.603548 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.603524    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"]
Jul 24 00:43:53.687593 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.687549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-socket\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.687795 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.687620    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53823641-86a5-4a46-baa0-5acc69491711-istio-podinfo\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.687795 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.687694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-envoy\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.687795 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.687726    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-certs\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.687795 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.687776    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53823641-86a5-4a46-baa0-5acc69491711-istiod-ca-cert\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.688020 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.687961    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-credential-socket\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.688020 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.688012    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf6c5\" (UniqueName: \"kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-kube-api-access-wf6c5\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.688127 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.688047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-istio-token\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.688127 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.688078    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-data\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.788752 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.788705    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-socket\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.788752 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.788749    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53823641-86a5-4a46-baa0-5acc69491711-istio-podinfo\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.788802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-envoy\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.788827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-certs\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.788865    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53823641-86a5-4a46-baa0-5acc69491711-istiod-ca-cert\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.788912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-credential-socket\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.788936    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf6c5\" (UniqueName: \"kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-kube-api-access-wf6c5\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.788964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-istio-token\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.788990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-data\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789360 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.789188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-socket\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789655 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.789624    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-certs\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789787 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.789682    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-credential-socket\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789787 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.789729    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53823641-86a5-4a46-baa0-5acc69491711-istiod-ca-cert\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.789787 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.789743    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-data\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.791281 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.791253    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-envoy\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.791406 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.791361    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53823641-86a5-4a46-baa0-5acc69491711-istio-podinfo\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.802054 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.802027    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-istio-token\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.803357 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.803331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf6c5\" (UniqueName: \"kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-kube-api-access-wf6c5\") pod \"e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") " pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:53.896500 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:53.896403    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:54.024780 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:54.024715    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"]
Jul 24 00:43:54.027479 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:43:54.027454    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53823641_86a5_4a46_baa0_5acc69491711.slice/crio-6699d3e491d2f904d386a761e3bed8b21888a0b7673827208ffb3b391ebedc12 WatchSource:0}: Error finding container 6699d3e491d2f904d386a761e3bed8b21888a0b7673827208ffb3b391ebedc12: Status 404 returned error can't find the container with id 6699d3e491d2f904d386a761e3bed8b21888a0b7673827208ffb3b391ebedc12
Jul 24 00:43:54.029536 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:54.029507    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:43:54.029595 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:54.029569    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:43:54.029629 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:54.029599    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:43:54.560892 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:54.560857    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" event={"ID":"53823641-86a5-4a46-baa0-5acc69491711","Type":"ContainerStarted","Data":"b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998"}
Jul 24 00:43:54.561084 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:54.560918    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" event={"ID":"53823641-86a5-4a46-baa0-5acc69491711","Type":"ContainerStarted","Data":"6699d3e491d2f904d386a761e3bed8b21888a0b7673827208ffb3b391ebedc12"}
Jul 24 00:43:54.584185 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:54.584133    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" podStartSLOduration=1.5841150910000001 podStartE2EDuration="1.584115091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:43:54.582573835 +0000 UTC m=+1997.430892576" watchObservedRunningTime="2026-07-24 00:43:54.584115091 +0000 UTC m=+1997.432433831"
Jul 24 00:43:54.897145 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:54.897098    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:55.901989 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:55.901959    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:56.568110 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.568074    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:56.569248 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.569228    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:43:56.619827 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.619790    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"]
Jul 24 00:43:56.620545 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.620521    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy"
Jul 24 00:43:56.620718 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.620706    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy"
Jul 24 00:43:56.620918 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.620895    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2288b1-50ff-444a-9313-49bd2218a3b0" containerName="istio-proxy"
Jul 24 00:43:56.623591 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.623567    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:56.626553 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.626283    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 00:43:56.626553 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.626404    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 00:43:56.626726 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.626554    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-464c3345-serving-cert\""
Jul 24 00:43:56.626726 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.626619    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nv7dj\""
Jul 24 00:43:56.632251 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.632221    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"]
Jul 24 00:43:56.713270 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.713234    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hblqs\" (UniqueName: \"kubernetes.io/projected/25bba1d1-a795-4f4b-afbe-6c44f065a826-kube-api-access-hblqs\") pod \"maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:56.713489 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.713371    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25bba1d1-a795-4f4b-afbe-6c44f065a826-maas-api-tls\") pod \"maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:56.713560 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.713509    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bba1d1-a795-4f4b-afbe-6c44f065a826-openshift-service-ca\") pod \"maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:56.814988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.814943    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bba1d1-a795-4f4b-afbe-6c44f065a826-openshift-service-ca\") pod \"maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:56.814988 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.814993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hblqs\" (UniqueName: \"kubernetes.io/projected/25bba1d1-a795-4f4b-afbe-6c44f065a826-kube-api-access-hblqs\") pod \"maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:56.815199 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.815029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25bba1d1-a795-4f4b-afbe-6c44f065a826-maas-api-tls\") pod \"maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:56.815649 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.815622    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bba1d1-a795-4f4b-afbe-6c44f065a826-openshift-service-ca\") pod \"maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:56.817485 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.817464    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25bba1d1-a795-4f4b-afbe-6c44f065a826-maas-api-tls\") pod \"maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:56.826266 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.826206    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hblqs\" (UniqueName: \"kubernetes.io/projected/25bba1d1-a795-4f4b-afbe-6c44f065a826-kube-api-access-hblqs\") pod \"maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:56.938901 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:56.938863    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:57.068768 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:57.068715    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"]
Jul 24 00:43:57.071307 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:43:57.071279    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bba1d1_a795_4f4b_afbe_6c44f065a826.slice/crio-e0b0b277227bdc02d37fbbb97cda47c9ae96c8187ceb1c6c23a0f9280cf9df91 WatchSource:0}: Error finding container e0b0b277227bdc02d37fbbb97cda47c9ae96c8187ceb1c6c23a0f9280cf9df91: Status 404 returned error can't find the container with id e0b0b277227bdc02d37fbbb97cda47c9ae96c8187ceb1c6c23a0f9280cf9df91
Jul 24 00:43:57.072703 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:57.072686    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:43:57.573548 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:57.573511    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n" event={"ID":"25bba1d1-a795-4f4b-afbe-6c44f065a826","Type":"ContainerStarted","Data":"81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002"}
Jul 24 00:43:57.573747 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:57.573574    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n" event={"ID":"25bba1d1-a795-4f4b-afbe-6c44f065a826","Type":"ContainerStarted","Data":"e0b0b277227bdc02d37fbbb97cda47c9ae96c8187ceb1c6c23a0f9280cf9df91"}
Jul 24 00:43:57.573747 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:57.573714    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:43:57.590993 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:43:57.590934    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n" podStartSLOduration=1.195951824 podStartE2EDuration="1.590911376s" totalImagesPullingTime="394.959552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:43:57.589168825 +0000 UTC m=+2000.437487566" watchObservedRunningTime="2026-07-24 00:43:57.590911376 +0000 UTC m=+2000.439230117"
Jul 24 00:44:03.584408 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:03.584369    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:44:09.626702 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.626673    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"]
Jul 24 00:44:09.630295 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.630277    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.633738 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.633708    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-464c3345\"/\"default-dockercfg-z6jc9\""
Jul 24 00:44:09.633866 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.633781    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-464c3345\"/\"kube-root-ca.crt\""
Jul 24 00:44:09.633866 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.633791    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-464c3345\"/\"e2e-lifecycle-model-464c3345-kserve-self-signed-certs\""
Jul 24 00:44:09.633866 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.633785    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-464c3345\"/\"openshift-service-ca.crt\""
Jul 24 00:44:09.638998 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.638968    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"]
Jul 24 00:44:09.726270 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.726237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hn6s\" (UniqueName: \"kubernetes.io/projected/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-kube-api-access-6hn6s\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.726475 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.726279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-dshm\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.726475 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.726304    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-home\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.726475 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.726390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tmp-dir\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.726475 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.726415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tls-certs\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.726657 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.726497    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-model-cache\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.827332 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.827265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tls-certs\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.827525 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.827383    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-model-cache\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.827525 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.827429    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hn6s\" (UniqueName: \"kubernetes.io/projected/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-kube-api-access-6hn6s\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.827525 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.827474    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-dshm\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.827525 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.827505    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-home\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.827745 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.827537    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tmp-dir\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.827915 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.827888    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-model-cache\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.828194 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.828166    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-home\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.828437 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.828416    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tmp-dir\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.830085 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.830062    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tls-certs\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.830423 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.830401    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-dshm\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.836292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.836262    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hn6s\" (UniqueName: \"kubernetes.io/projected/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-kube-api-access-6hn6s\") pod \"e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") " pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:09.941275 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:09.941185    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:10.078908 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:10.078884    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"]
Jul 24 00:44:10.081430 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:44:10.081390    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03eeaf0b_82e3_4fcf_a4d1_eec412254a76.slice/crio-0a424e82f4e8fc972fe7f21015c720a9ecfd9ff763a61ff95192648011164d4e WatchSource:0}: Error finding container 0a424e82f4e8fc972fe7f21015c720a9ecfd9ff763a61ff95192648011164d4e: Status 404 returned error can't find the container with id 0a424e82f4e8fc972fe7f21015c720a9ecfd9ff763a61ff95192648011164d4e
Jul 24 00:44:10.630921 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:10.630891    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5" event={"ID":"03eeaf0b-82e3-4fcf-a4d1-eec412254a76","Type":"ContainerStarted","Data":"eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190"}
Jul 24 00:44:10.631325 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:10.630933    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5" event={"ID":"03eeaf0b-82e3-4fcf-a4d1-eec412254a76","Type":"ContainerStarted","Data":"0a424e82f4e8fc972fe7f21015c720a9ecfd9ff763a61ff95192648011164d4e"}
Jul 24 00:44:10.632547 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:10.632521    2574 generic.go:356] "Generic (PLEG): container finished" podID="53823641-86a5-4a46-baa0-5acc69491711" containerID="b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998" exitCode=0
Jul 24 00:44:10.632649 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:10.632567    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" event={"ID":"53823641-86a5-4a46-baa0-5acc69491711","Type":"ContainerDied","Data":"b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998"}
Jul 24 00:44:10.632966 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:10.632950    2574 scope.go:117] "RemoveContainer" containerID="b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998"
Jul 24 00:44:10.633927 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:10.633900    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:44:10.634021 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:10.633983    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:44:10.634081 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:10.634033    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:44:10.648077 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:10.648039    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5" podStartSLOduration=1.648025233 podStartE2EDuration="1.648025233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:44:10.646784521 +0000 UTC m=+2013.495103264" watchObservedRunningTime="2026-07-24 00:44:10.648025233 +0000 UTC m=+2013.496343970"
Jul 24 00:44:11.568128 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:11.568078    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:44:11.639250 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:11.639215    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" event={"ID":"53823641-86a5-4a46-baa0-5acc69491711","Type":"ContainerStarted","Data":"6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33"}
Jul 24 00:44:11.896950 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:11.896853    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:44:11.898095 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:11.898069    2574 patch_prober.go:28] interesting pod/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 24 00:44:11.898228 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:11.898128    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 24 00:44:12.897602 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:12.897567    2574 patch_prober.go:28] interesting pod/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 24 00:44:12.897981 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:12.897625    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 24 00:44:13.897163 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:13.897124    2574 patch_prober.go:28] interesting pod/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 24 00:44:13.897368 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:13.897183    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 24 00:44:14.897521 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:14.897484    2574 patch_prober.go:28] interesting pod/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 24 00:44:14.897905 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:14.897543    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 24 00:44:15.900848 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:15.900816    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:44:15.901234 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:15.901082    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:44:15.901718 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:15.901698    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:44:19.941941 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:19.941900    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:19.942437 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:19.941956    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:19.954515 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:19.954490    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:20.708233 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:20.708203    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:44:46.077763 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.077731    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"]
Jul 24 00:44:46.078227 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.077939    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n" podUID="25bba1d1-a795-4f4b-afbe-6c44f065a826" containerName="maas-api" containerID="cri-o://81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002" gracePeriod=30
Jul 24 00:44:46.326432 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.326408    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:44:46.449649 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.449561    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25bba1d1-a795-4f4b-afbe-6c44f065a826-maas-api-tls\") pod \"25bba1d1-a795-4f4b-afbe-6c44f065a826\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") "
Jul 24 00:44:46.449649 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.449593    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bba1d1-a795-4f4b-afbe-6c44f065a826-openshift-service-ca\") pod \"25bba1d1-a795-4f4b-afbe-6c44f065a826\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") "
Jul 24 00:44:46.449649 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.449633    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hblqs\" (UniqueName: \"kubernetes.io/projected/25bba1d1-a795-4f4b-afbe-6c44f065a826-kube-api-access-hblqs\") pod \"25bba1d1-a795-4f4b-afbe-6c44f065a826\" (UID: \"25bba1d1-a795-4f4b-afbe-6c44f065a826\") "
Jul 24 00:44:46.450019 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.449989    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25bba1d1-a795-4f4b-afbe-6c44f065a826-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25bba1d1-a795-4f4b-afbe-6c44f065a826" (UID: "25bba1d1-a795-4f4b-afbe-6c44f065a826"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:44:46.451757 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.451729    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25bba1d1-a795-4f4b-afbe-6c44f065a826-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25bba1d1-a795-4f4b-afbe-6c44f065a826" (UID: "25bba1d1-a795-4f4b-afbe-6c44f065a826"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:44:46.451883 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.451764    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bba1d1-a795-4f4b-afbe-6c44f065a826-kube-api-access-hblqs" (OuterVolumeSpecName: "kube-api-access-hblqs") pod "25bba1d1-a795-4f4b-afbe-6c44f065a826" (UID: "25bba1d1-a795-4f4b-afbe-6c44f065a826"). InnerVolumeSpecName "kube-api-access-hblqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:44:46.550849 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.550821    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25bba1d1-a795-4f4b-afbe-6c44f065a826-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:44:46.550849 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.550847    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bba1d1-a795-4f4b-afbe-6c44f065a826-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:44:46.551013 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.550857    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hblqs\" (UniqueName: \"kubernetes.io/projected/25bba1d1-a795-4f4b-afbe-6c44f065a826-kube-api-access-hblqs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:44:46.798599 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.798522    2574 generic.go:356] "Generic (PLEG): container finished" podID="25bba1d1-a795-4f4b-afbe-6c44f065a826" containerID="81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002" exitCode=0
Jul 24 00:44:46.798599 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.798573    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"
Jul 24 00:44:46.798758 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.798606    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n" event={"ID":"25bba1d1-a795-4f4b-afbe-6c44f065a826","Type":"ContainerDied","Data":"81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002"}
Jul 24 00:44:46.798758 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.798659    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n" event={"ID":"25bba1d1-a795-4f4b-afbe-6c44f065a826","Type":"ContainerDied","Data":"e0b0b277227bdc02d37fbbb97cda47c9ae96c8187ceb1c6c23a0f9280cf9df91"}
Jul 24 00:44:46.798758 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.798681    2574 scope.go:117] "RemoveContainer" containerID="81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002"
Jul 24 00:44:46.807031 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.807014    2574 scope.go:117] "RemoveContainer" containerID="81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002"
Jul 24 00:44:46.807273 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:44:46.807253    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002\": container with ID starting with 81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002 not found: ID does not exist" containerID="81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002"
Jul 24 00:44:46.807345 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.807281    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002"} err="failed to get container status \"81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002\": rpc error: code = NotFound desc = could not find container \"81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002\": container with ID starting with 81516c1796c8f9b776d177afc5db5c08b53b56b58d737816fba1f6608dccf002 not found: ID does not exist"
Jul 24 00:44:46.818201 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.818173    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"]
Jul 24 00:44:46.820346 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:46.820310    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-464c3345-7d4d475c6-9gm4n"]
Jul 24 00:44:47.752310 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:47.752275    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25bba1d1-a795-4f4b-afbe-6c44f065a826" path="/var/lib/kubelet/pods/25bba1d1-a795-4f4b-afbe-6c44f065a826/volumes"
Jul 24 00:44:59.089931 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:44:59.089897    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"]
Jul 24 00:45:14.108728 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.108663    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5" podUID="03eeaf0b-82e3-4fcf-a4d1-eec412254a76" containerName="main" containerID="cri-o://eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190" gracePeriod=45
Jul 24 00:45:14.359778 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.359715    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:45:14.460550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.460508    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-home\") pod \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") "
Jul 24 00:45:14.460550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.460553    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tmp-dir\") pod \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") "
Jul 24 00:45:14.460800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.460582    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-model-cache\") pod \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") "
Jul 24 00:45:14.460800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.460655    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hn6s\" (UniqueName: \"kubernetes.io/projected/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-kube-api-access-6hn6s\") pod \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") "
Jul 24 00:45:14.460800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.460716    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-dshm\") pod \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") "
Jul 24 00:45:14.460800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.460738    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tls-certs\") pod \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\" (UID: \"03eeaf0b-82e3-4fcf-a4d1-eec412254a76\") "
Jul 24 00:45:14.460996 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.460780    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-home" (OuterVolumeSpecName: "home") pod "03eeaf0b-82e3-4fcf-a4d1-eec412254a76" (UID: "03eeaf0b-82e3-4fcf-a4d1-eec412254a76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:45:14.460996 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.460904    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03eeaf0b-82e3-4fcf-a4d1-eec412254a76" (UID: "03eeaf0b-82e3-4fcf-a4d1-eec412254a76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:45:14.460996 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.460925    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-model-cache" (OuterVolumeSpecName: "model-cache") pod "03eeaf0b-82e3-4fcf-a4d1-eec412254a76" (UID: "03eeaf0b-82e3-4fcf-a4d1-eec412254a76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:45:14.461156 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.461041    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.461156 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.461076    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.461156 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.461087    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.462948 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.462920    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03eeaf0b-82e3-4fcf-a4d1-eec412254a76" (UID: "03eeaf0b-82e3-4fcf-a4d1-eec412254a76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:45:14.463088 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.463021    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-dshm" (OuterVolumeSpecName: "dshm") pod "03eeaf0b-82e3-4fcf-a4d1-eec412254a76" (UID: "03eeaf0b-82e3-4fcf-a4d1-eec412254a76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:45:14.463088 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.463044    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-kube-api-access-6hn6s" (OuterVolumeSpecName: "kube-api-access-6hn6s") pod "03eeaf0b-82e3-4fcf-a4d1-eec412254a76" (UID: "03eeaf0b-82e3-4fcf-a4d1-eec412254a76"). InnerVolumeSpecName "kube-api-access-6hn6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:45:14.562380 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.562349    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.562380 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.562375    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.562380 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.562385    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hn6s\" (UniqueName: \"kubernetes.io/projected/03eeaf0b-82e3-4fcf-a4d1-eec412254a76-kube-api-access-6hn6s\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.910650 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.910613    2574 generic.go:356] "Generic (PLEG): container finished" podID="03eeaf0b-82e3-4fcf-a4d1-eec412254a76" containerID="eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190" exitCode=0
Jul 24 00:45:14.910650 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.910652    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"
Jul 24 00:45:14.910872 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.910703    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5" event={"ID":"03eeaf0b-82e3-4fcf-a4d1-eec412254a76","Type":"ContainerDied","Data":"eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190"}
Jul 24 00:45:14.910872 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.910742    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5" event={"ID":"03eeaf0b-82e3-4fcf-a4d1-eec412254a76","Type":"ContainerDied","Data":"0a424e82f4e8fc972fe7f21015c720a9ecfd9ff763a61ff95192648011164d4e"}
Jul 24 00:45:14.910872 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.910763    2574 scope.go:117] "RemoveContainer" containerID="eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190"
Jul 24 00:45:14.919301 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.919284    2574 scope.go:117] "RemoveContainer" containerID="eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190"
Jul 24 00:45:14.919553 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:45:14.919536    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190\": container with ID starting with eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190 not found: ID does not exist" containerID="eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190"
Jul 24 00:45:14.919609 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.919561    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190"} err="failed to get container status \"eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190\": rpc error: code = NotFound desc = could not find container \"eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190\": container with ID starting with eb7f19922e91ee296db22aaf5c571df7242c1a868aa49467dfefea3192254190 not found: ID does not exist"
Jul 24 00:45:14.932169 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.932145    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"]
Jul 24 00:45:14.936365 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:14.936345    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-464c3345/e2e-lifecycle-model-464c3345-kserve-7d9fcf8cdc-fj9d5"]
Jul 24 00:45:15.752369 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:15.752336    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03eeaf0b-82e3-4fcf-a4d1-eec412254a76" path="/var/lib/kubelet/pods/03eeaf0b-82e3-4fcf-a4d1-eec412254a76/volumes"
Jul 24 00:45:42.236684 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:42.236649    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"]
Jul 24 00:45:42.237183 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:42.236844    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy" containerID="cri-o://6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33" gracePeriod=30
Jul 24 00:45:45.903063 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:45.903024    2574 patch_prober.go:28] interesting pod/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 00:45:45.903479 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:45.903089    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 00:45:47.490043 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.490015    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:45:47.622188 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.622150    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-data\") pod \"53823641-86a5-4a46-baa0-5acc69491711\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") "
Jul 24 00:45:47.622411 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.622197    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53823641-86a5-4a46-baa0-5acc69491711-istio-podinfo\") pod \"53823641-86a5-4a46-baa0-5acc69491711\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") "
Jul 24 00:45:47.622411 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.622235    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-envoy\") pod \"53823641-86a5-4a46-baa0-5acc69491711\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") "
Jul 24 00:45:47.622411 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.622273    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-istio-token\") pod \"53823641-86a5-4a46-baa0-5acc69491711\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") "
Jul 24 00:45:47.622411 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.622308    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53823641-86a5-4a46-baa0-5acc69491711-istiod-ca-cert\") pod \"53823641-86a5-4a46-baa0-5acc69491711\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") "
Jul 24 00:45:47.622411 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.622368    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-certs\") pod \"53823641-86a5-4a46-baa0-5acc69491711\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") "
Jul 24 00:45:47.622411 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.622391    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf6c5\" (UniqueName: \"kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-kube-api-access-wf6c5\") pod \"53823641-86a5-4a46-baa0-5acc69491711\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") "
Jul 24 00:45:47.622740 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.622431    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-socket\") pod \"53823641-86a5-4a46-baa0-5acc69491711\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") "
Jul 24 00:45:47.622740 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.622495    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-credential-socket\") pod \"53823641-86a5-4a46-baa0-5acc69491711\" (UID: \"53823641-86a5-4a46-baa0-5acc69491711\") "
Jul 24 00:45:47.622844 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.622794    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "53823641-86a5-4a46-baa0-5acc69491711" (UID: "53823641-86a5-4a46-baa0-5acc69491711"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:45:47.623115 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.622917    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "53823641-86a5-4a46-baa0-5acc69491711" (UID: "53823641-86a5-4a46-baa0-5acc69491711"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:45:47.623115 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.623070    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-data" (OuterVolumeSpecName: "istio-data") pod "53823641-86a5-4a46-baa0-5acc69491711" (UID: "53823641-86a5-4a46-baa0-5acc69491711"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:45:47.623115 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.623081    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "53823641-86a5-4a46-baa0-5acc69491711" (UID: "53823641-86a5-4a46-baa0-5acc69491711"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:45:47.623115 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.623074    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53823641-86a5-4a46-baa0-5acc69491711-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "53823641-86a5-4a46-baa0-5acc69491711" (UID: "53823641-86a5-4a46-baa0-5acc69491711"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:45:47.624603 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.624576    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53823641-86a5-4a46-baa0-5acc69491711-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "53823641-86a5-4a46-baa0-5acc69491711" (UID: "53823641-86a5-4a46-baa0-5acc69491711"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 00:45:47.624815 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.624785    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "53823641-86a5-4a46-baa0-5acc69491711" (UID: "53823641-86a5-4a46-baa0-5acc69491711"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:45:47.624954 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.624932    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-istio-token" (OuterVolumeSpecName: "istio-token") pod "53823641-86a5-4a46-baa0-5acc69491711" (UID: "53823641-86a5-4a46-baa0-5acc69491711"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:45:47.625287 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.625255    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-kube-api-access-wf6c5" (OuterVolumeSpecName: "kube-api-access-wf6c5") pod "53823641-86a5-4a46-baa0-5acc69491711" (UID: "53823641-86a5-4a46-baa0-5acc69491711"). InnerVolumeSpecName "kube-api-access-wf6c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:45:47.724016 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.723980    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-envoy\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:47.724016 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.724007    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-istio-token\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:47.724016 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.724015    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53823641-86a5-4a46-baa0-5acc69491711-istiod-ca-cert\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:47.724016 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.724024    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:47.724397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.724032    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf6c5\" (UniqueName: \"kubernetes.io/projected/53823641-86a5-4a46-baa0-5acc69491711-kube-api-access-wf6c5\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:47.724397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.724043    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-workload-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:47.724397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.724051    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-credential-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:47.724397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.724059    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53823641-86a5-4a46-baa0-5acc69491711-istio-data\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:47.724397 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:47.724068    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53823641-86a5-4a46-baa0-5acc69491711-istio-podinfo\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:45:48.049214 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.049136    2574 generic.go:356] "Generic (PLEG): container finished" podID="53823641-86a5-4a46-baa0-5acc69491711" containerID="6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33" exitCode=0
Jul 24 00:45:48.049214 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.049181    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"
Jul 24 00:45:48.049214 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.049200    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" event={"ID":"53823641-86a5-4a46-baa0-5acc69491711","Type":"ContainerDied","Data":"6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33"}
Jul 24 00:45:48.049467 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.049245    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp" event={"ID":"53823641-86a5-4a46-baa0-5acc69491711","Type":"ContainerDied","Data":"6699d3e491d2f904d386a761e3bed8b21888a0b7673827208ffb3b391ebedc12"}
Jul 24 00:45:48.049467 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.049266    2574 scope.go:117] "RemoveContainer" containerID="6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33"
Jul 24 00:45:48.057459 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.057434    2574 scope.go:117] "RemoveContainer" containerID="b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998"
Jul 24 00:45:48.065171 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.065153    2574 scope.go:117] "RemoveContainer" containerID="6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33"
Jul 24 00:45:48.065469 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:45:48.065445    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33\": container with ID starting with 6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33 not found: ID does not exist" containerID="6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33"
Jul 24 00:45:48.065537 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.065477    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33"} err="failed to get container status \"6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33\": rpc error: code = NotFound desc = could not find container \"6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33\": container with ID starting with 6a33226a3de5a4c923d36dc46e4ea5023609c5de7db8b3408ea78bfae7f40d33 not found: ID does not exist"
Jul 24 00:45:48.065537 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.065493    2574 scope.go:117] "RemoveContainer" containerID="b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998"
Jul 24 00:45:48.065805 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:45:48.065767    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998\": container with ID starting with b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998 not found: ID does not exist" containerID="b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998"
Jul 24 00:45:48.065920 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.065810    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998"} err="failed to get container status \"b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998\": rpc error: code = NotFound desc = could not find container \"b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998\": container with ID starting with b1e3acb1e78a6c4b77ade71dcad22859919c2bddfc6d4eba3d8f081dd9357998 not found: ID does not exist"
Jul 24 00:45:48.067862 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.067841    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"]
Jul 24 00:45:48.071367 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:48.071345    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-464c3345-openshift-default-7f975ccf7b-9x5mp"]
Jul 24 00:45:49.752403 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:45:49.752370    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53823641-86a5-4a46-baa0-5acc69491711" path="/var/lib/kubelet/pods/53823641-86a5-4a46-baa0-5acc69491711/volumes"
Jul 24 00:46:38.610787 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.610750    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"]
Jul 24 00:46:38.611191 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611174    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy"
Jul 24 00:46:38.611191 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611188    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy"
Jul 24 00:46:38.611264 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611205    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy"
Jul 24 00:46:38.611264 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611213    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy"
Jul 24 00:46:38.611264 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611225    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25bba1d1-a795-4f4b-afbe-6c44f065a826" containerName="maas-api"
Jul 24 00:46:38.611264 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611233    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="25bba1d1-a795-4f4b-afbe-6c44f065a826" containerName="maas-api"
Jul 24 00:46:38.611264 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611242    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03eeaf0b-82e3-4fcf-a4d1-eec412254a76" containerName="main"
Jul 24 00:46:38.611264 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611249    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="03eeaf0b-82e3-4fcf-a4d1-eec412254a76" containerName="main"
Jul 24 00:46:38.611475 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611357    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="25bba1d1-a795-4f4b-afbe-6c44f065a826" containerName="maas-api"
Jul 24 00:46:38.611475 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611368    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy"
Jul 24 00:46:38.611475 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611379    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="53823641-86a5-4a46-baa0-5acc69491711" containerName="istio-proxy"
Jul 24 00:46:38.611475 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.611390    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="03eeaf0b-82e3-4fcf-a4d1-eec412254a76" containerName="main"
Jul 24 00:46:38.615721 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.615701    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.618338 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.618291    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c28612-openshift-default-dockercfg-gz2zg\""
Jul 24 00:46:38.626614 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.626566    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"]
Jul 24 00:46:38.738394 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.738357    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-socket\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.738394 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.738394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-podinfo\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.738608 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.738416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcbfb\" (UniqueName: \"kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-kube-api-access-wcbfb\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.738608 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.738529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istiod-ca-cert\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.738608 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.738586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-envoy\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.738711 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.738628    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-credential-socket\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.738711 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.738657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-token\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.738711 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.738692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-certs\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.738807 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.738718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-data\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.839513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.839472    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istiod-ca-cert\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.839513 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.839521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-envoy\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.839764 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.839554    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-credential-socket\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.839764 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.839573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-token\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.839764 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.839597    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-certs\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.839764 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.839622    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-data\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.839764 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.839684    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-socket\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.839764 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.839722    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-podinfo\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.839764 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.839754    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcbfb\" (UniqueName: \"kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-kube-api-access-wcbfb\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.840122 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.839986    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-data\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.840122 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.840083    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-certs\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.840223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.840164    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-socket\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.840353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.840307    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istiod-ca-cert\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.840481 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.840349    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-credential-socket\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.841891 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.841869    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-envoy\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.842238 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.842217    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-podinfo\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.849164 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.849139    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-token\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.849698 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.849674    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcbfb\" (UniqueName: \"kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-kube-api-access-wcbfb\") pod \"e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") " pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:38.926895 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:38.926813    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:39.061024 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:39.060991    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"]
Jul 24 00:46:39.064112 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:46:39.064087    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a8aff7_c16d_4374_8ebe_8bc455ee3aa1.slice/crio-0f2750429d43faef9f413f940409f6248b84d25becea547188378457f46d5332 WatchSource:0}: Error finding container 0f2750429d43faef9f413f940409f6248b84d25becea547188378457f46d5332: Status 404 returned error can't find the container with id 0f2750429d43faef9f413f940409f6248b84d25becea547188378457f46d5332
Jul 24 00:46:39.066082 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:39.066051    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:46:39.066156 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:39.066116    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:46:39.066156 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:39.066147    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:46:39.260060 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:39.259966    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" event={"ID":"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1","Type":"ContainerStarted","Data":"65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252"}
Jul 24 00:46:39.260060 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:39.260046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" event={"ID":"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1","Type":"ContainerStarted","Data":"0f2750429d43faef9f413f940409f6248b84d25becea547188378457f46d5332"}
Jul 24 00:46:39.279994 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:39.279930    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" podStartSLOduration=1.279909118 podStartE2EDuration="1.279909118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:46:39.279032822 +0000 UTC m=+2162.127351565" watchObservedRunningTime="2026-07-24 00:46:39.279909118 +0000 UTC m=+2162.128227859"
Jul 24 00:46:39.927653 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:39.927612    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:40.932518 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:40.932488    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:41.269086 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.269002    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:41.270030 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.270014    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:41.571045 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.571011    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"]
Jul 24 00:46:41.574275 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.574260    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:41.583583 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.583562    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c28612-serving-cert\""
Jul 24 00:46:41.583741 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.583722    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nv7dj\""
Jul 24 00:46:41.584041 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.584027    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 00:46:41.587345 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.587308    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"]
Jul 24 00:46:41.591751 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.591733    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 00:46:41.664001 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.663970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:41.664194 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.664037    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:41.664194 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.664083    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8969q\" (UniqueName: \"kubernetes.io/projected/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-kube-api-access-8969q\") pod \"maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:41.765358 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.765304    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:41.765507 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.765397    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:41.765507 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.765423    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8969q\" (UniqueName: \"kubernetes.io/projected/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-kube-api-access-8969q\") pod \"maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:41.766020 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.766002    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:41.767690 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.767668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:41.784279 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.784249    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8969q\" (UniqueName: \"kubernetes.io/projected/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-kube-api-access-8969q\") pod \"maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:41.884596 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:41.884517    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:42.026334 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:42.026287    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"]
Jul 24 00:46:42.029225 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:46:42.029195    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7044fd5d_1dbf_4337_9189_e1bd5dea99fe.slice/crio-7be1a3a97e20e290bbc089cb845d540f05316dfb0de85e61d5a66024fd302fe7 WatchSource:0}: Error finding container 7be1a3a97e20e290bbc089cb845d540f05316dfb0de85e61d5a66024fd302fe7: Status 404 returned error can't find the container with id 7be1a3a97e20e290bbc089cb845d540f05316dfb0de85e61d5a66024fd302fe7
Jul 24 00:46:42.274176 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:42.274095    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr" event={"ID":"7044fd5d-1dbf-4337-9189-e1bd5dea99fe","Type":"ContainerStarted","Data":"7be1a3a97e20e290bbc089cb845d540f05316dfb0de85e61d5a66024fd302fe7"}
Jul 24 00:46:43.279692 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:43.279657    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr" event={"ID":"7044fd5d-1dbf-4337-9189-e1bd5dea99fe","Type":"ContainerStarted","Data":"62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49"}
Jul 24 00:46:43.305985 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:43.305934    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr" podStartSLOduration=1.863214301 podStartE2EDuration="2.30591955s" totalImagesPullingTime="442.705249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:46:43.304642299 +0000 UTC m=+2166.152961039" watchObservedRunningTime="2026-07-24 00:46:43.30591955 +0000 UTC m=+2166.154238349"
Jul 24 00:46:44.283682 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:44.283653    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:49.296371 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.296338    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"]
Jul 24 00:46:49.299713 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.299677    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:49.306441 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.306415    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5f040e-serving-cert\""
Jul 24 00:46:49.318409 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.318381    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"]
Jul 24 00:46:49.429015 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.428980    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:49.429192 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.429038    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:49.429192 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.429087    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fblsm\" (UniqueName: \"kubernetes.io/projected/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-kube-api-access-fblsm\") pod \"maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:49.530023 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.529982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fblsm\" (UniqueName: \"kubernetes.io/projected/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-kube-api-access-fblsm\") pod \"maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:49.530241 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.530067    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:49.530241 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.530119    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:49.530241 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:46:49.530219    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-serving-cert: secret "maas-api-e2e-inf-b-5f040e-serving-cert" not found
Jul 24 00:46:49.530392 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:46:49.530285    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-maas-api-tls podName:8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2 nodeName:}" failed. No retries permitted until 2026-07-24 00:46:50.030265896 +0000 UTC m=+2172.878584622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-maas-api-tls") pod "maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp" (UID: "8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2") : secret "maas-api-e2e-inf-b-5f040e-serving-cert" not found
Jul 24 00:46:49.530701 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.530680    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:49.539627 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:49.539601    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fblsm\" (UniqueName: \"kubernetes.io/projected/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-kube-api-access-fblsm\") pod \"maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:50.033913 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:50.033880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:50.036297 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:50.036266    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:50.211021 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:50.210980    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:50.295420 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:50.295156    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:46:50.588441 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:50.588415    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"]
Jul 24 00:46:50.590792 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:46:50.590768    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a76dc31_c4a9_433d_9fbe_1f6cb45b4bc2.slice/crio-0268e3d6d0fce8236e7f79e3571ad57e7c8223c50c6970c8d1b6904516c4bbd9 WatchSource:0}: Error finding container 0268e3d6d0fce8236e7f79e3571ad57e7c8223c50c6970c8d1b6904516c4bbd9: Status 404 returned error can't find the container with id 0268e3d6d0fce8236e7f79e3571ad57e7c8223c50c6970c8d1b6904516c4bbd9
Jul 24 00:46:51.315378 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:51.315295    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp" event={"ID":"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2","Type":"ContainerStarted","Data":"d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d"}
Jul 24 00:46:51.315378 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:51.315364    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp" event={"ID":"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2","Type":"ContainerStarted","Data":"0268e3d6d0fce8236e7f79e3571ad57e7c8223c50c6970c8d1b6904516c4bbd9"}
Jul 24 00:46:51.315607 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:51.315435    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:51.336541 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:51.336491    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp" podStartSLOduration=1.994735476 podStartE2EDuration="2.33647478s" totalImagesPullingTime="341.739304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:46:51.335630026 +0000 UTC m=+2174.183948770" watchObservedRunningTime="2026-07-24 00:46:51.33647478 +0000 UTC m=+2174.184793541"
Jul 24 00:46:54.328966 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:54.328884    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerID="65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252" exitCode=0
Jul 24 00:46:54.329489 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:54.328963    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" event={"ID":"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1","Type":"ContainerDied","Data":"65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252"}
Jul 24 00:46:54.329489 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:54.329479    2574 scope.go:117] "RemoveContainer" containerID="65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252"
Jul 24 00:46:54.330491 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:54.330467    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:46:54.330552 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:54.330532    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:46:54.330588 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:54.330575    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:46:55.338185 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:55.338147    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" event={"ID":"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1","Type":"ContainerStarted","Data":"e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5"}
Jul 24 00:46:55.927670 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:55.927632    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:55.928799 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:55.928774    2574 patch_prober.go:28] interesting pod/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 24 00:46:55.928882 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:55.928836    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 24 00:46:56.269103 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:56.269009    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:56.927600 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:56.927566    2574 patch_prober.go:28] interesting pod/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 24 00:46:56.927975 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:56.927624    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 24 00:46:57.325834 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:57.325804    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:46:57.927978 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:57.927938    2574 patch_prober.go:28] interesting pod/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 24 00:46:57.928400 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:57.927996    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 24 00:46:58.927925 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:58.927888    2574 patch_prober.go:28] interesting pod/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 24 00:46:58.928119 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:58.927963    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 24 00:46:59.526493 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.526450    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"]
Jul 24 00:46:59.529584 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.529565    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.533048 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.533024    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c28612\"/\"default-dockercfg-xwlr8\""
Jul 24 00:46:59.533187 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.533073    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c28612\"/\"kube-root-ca.crt\""
Jul 24 00:46:59.533187 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.533097    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c28612\"/\"openshift-service-ca.crt\""
Jul 24 00:46:59.533187 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.533073    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c28612\"/\"test-model-c28612-kserve-self-signed-certs\""
Jul 24 00:46:59.538023 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.538003    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"]
Jul 24 00:46:59.615038 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.615003    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tmp-dir\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.615221 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.615047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-model-cache\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.615221 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.615159    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-dshm\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.615221 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.615218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-home\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.615376 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.615247    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tls-certs\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.615376 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.615264    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r966g\" (UniqueName: \"kubernetes.io/projected/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-kube-api-access-r966g\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.716644 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.716608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-home\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.716833 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.716651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tls-certs\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.716833 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.716670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r966g\" (UniqueName: \"kubernetes.io/projected/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-kube-api-access-r966g\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.716965 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.716851    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tmp-dir\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.716965 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.716902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-model-cache\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.717130 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.716971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-dshm\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.717130 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.717039    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-home\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.717208 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.717165    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tmp-dir\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.717273 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.717255    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-model-cache\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.719106 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.719075    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-dshm\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.719237 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.719223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tls-certs\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.724675 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.724651    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r966g\" (UniqueName: \"kubernetes.io/projected/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-kube-api-access-r966g\") pod \"test-model-c28612-kserve-7457bf65fc-s64k5\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") " pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.839888 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.839853    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:46:59.931978 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.931946    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:59.933089 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.933068    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:46:59.974308 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:46:59.974189    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"]
Jul 24 00:47:00.366951 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:00.366916    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5" event={"ID":"56b8004d-9a8d-4e99-bacb-7435b1ffb80b","Type":"ContainerStarted","Data":"bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3"}
Jul 24 00:47:00.367149 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:00.366960    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5" event={"ID":"56b8004d-9a8d-4e99-bacb-7435b1ffb80b","Type":"ContainerStarted","Data":"8ee3c7ce324af62766773900dd55775c818f3335448f62621f37c9d2b28628d2"}
Jul 24 00:47:00.384698 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:00.384648    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5" podStartSLOduration=1.38463165 podStartE2EDuration="1.38463165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:47:00.383136476 +0000 UTC m=+2183.231455219" watchObservedRunningTime="2026-07-24 00:47:00.38463165 +0000 UTC m=+2183.232950414"
Jul 24 00:47:09.840130 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:09.840067    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:47:09.840731 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:09.840152    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:47:09.852614 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:09.852588    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:47:10.426138 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:10.426109    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:47:28.116802 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.116769    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"]
Jul 24 00:47:28.120126 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.120107    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.123823 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.123779    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5f040e\"/\"test-model-5f040e-kserve-self-signed-certs\""
Jul 24 00:47:28.123981 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.123779    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5f040e\"/\"default-dockercfg-d8ccv\""
Jul 24 00:47:28.123981 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.123794    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5f040e\"/\"kube-root-ca.crt\""
Jul 24 00:47:28.124139 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.124071    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5f040e\"/\"openshift-service-ca.crt\""
Jul 24 00:47:28.128957 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.128935    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"]
Jul 24 00:47:28.164669 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.164638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6xxk\" (UniqueName: \"kubernetes.io/projected/a3338c9f-0148-48ad-83eb-d51205641a21-kube-api-access-j6xxk\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.164816 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.164683    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-tmp-dir\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.164816 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.164716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3338c9f-0148-48ad-83eb-d51205641a21-tls-certs\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.164816 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.164745    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-home\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.164816 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.164803    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-dshm\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.165012 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.164894    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-model-cache\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.266182 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.266143    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-dshm\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.266381 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.266210    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-model-cache\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.266381 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.266240    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6xxk\" (UniqueName: \"kubernetes.io/projected/a3338c9f-0148-48ad-83eb-d51205641a21-kube-api-access-j6xxk\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.266381 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.266271    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-tmp-dir\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.266381 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.266293    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3338c9f-0148-48ad-83eb-d51205641a21-tls-certs\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.266381 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.266334    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-home\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.266695 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.266669    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-model-cache\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.266762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.266698    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-home\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.266762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.266728    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-tmp-dir\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.268446 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.268422    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-dshm\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.268613 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.268595    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3338c9f-0148-48ad-83eb-d51205641a21-tls-certs\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.274438 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.274418    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6xxk\" (UniqueName: \"kubernetes.io/projected/a3338c9f-0148-48ad-83eb-d51205641a21-kube-api-access-j6xxk\") pod \"test-model-5f040e-kserve-584488bbb5-fss64\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") " pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.430893 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.430793    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:28.558612 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:28.558587    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"]
Jul 24 00:47:28.561417 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:47:28.561372    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3338c9f_0148_48ad_83eb_d51205641a21.slice/crio-3e98376b0fd2e1db2e367d883dfcd65cdca7d1fc55fe3d6262f0c06c089f6149 WatchSource:0}: Error finding container 3e98376b0fd2e1db2e367d883dfcd65cdca7d1fc55fe3d6262f0c06c089f6149: Status 404 returned error can't find the container with id 3e98376b0fd2e1db2e367d883dfcd65cdca7d1fc55fe3d6262f0c06c089f6149
Jul 24 00:47:29.498041 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:29.498002    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64" event={"ID":"a3338c9f-0148-48ad-83eb-d51205641a21","Type":"ContainerStarted","Data":"7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c"}
Jul 24 00:47:29.498449 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:29.498051    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64" event={"ID":"a3338c9f-0148-48ad-83eb-d51205641a21","Type":"ContainerStarted","Data":"3e98376b0fd2e1db2e367d883dfcd65cdca7d1fc55fe3d6262f0c06c089f6149"}
Jul 24 00:47:29.513557 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:29.513511    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64" podStartSLOduration=1.513496535 podStartE2EDuration="1.513496535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:47:29.512864388 +0000 UTC m=+2212.361183131" watchObservedRunningTime="2026-07-24 00:47:29.513496535 +0000 UTC m=+2212.361815275"
Jul 24 00:47:38.431825 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:38.431772    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:38.432405 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:38.431840    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:38.449618 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:38.449585    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:47:38.548566 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:47:38.548528    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:48:08.189495 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:08.189459    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"]
Jul 24 00:48:09.141159 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:09.141127    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"]
Jul 24 00:48:19.888274 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:19.888240    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"]
Jul 24 00:48:19.888728 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:19.888457    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr" podUID="7044fd5d-1dbf-4337-9189-e1bd5dea99fe" containerName="maas-api" containerID="cri-o://62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49" gracePeriod=30
Jul 24 00:48:20.134796 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.134771    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:48:20.236007 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.235920    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-maas-api-tls\") pod \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") "
Jul 24 00:48:20.236007 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.235968    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-openshift-service-ca\") pod \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") "
Jul 24 00:48:20.236007 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.236004    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8969q\" (UniqueName: \"kubernetes.io/projected/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-kube-api-access-8969q\") pod \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\" (UID: \"7044fd5d-1dbf-4337-9189-e1bd5dea99fe\") "
Jul 24 00:48:20.236377 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.236353    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7044fd5d-1dbf-4337-9189-e1bd5dea99fe" (UID: "7044fd5d-1dbf-4337-9189-e1bd5dea99fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:48:20.238100 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.238073    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-kube-api-access-8969q" (OuterVolumeSpecName: "kube-api-access-8969q") pod "7044fd5d-1dbf-4337-9189-e1bd5dea99fe" (UID: "7044fd5d-1dbf-4337-9189-e1bd5dea99fe"). InnerVolumeSpecName "kube-api-access-8969q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:48:20.238185 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.238083    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7044fd5d-1dbf-4337-9189-e1bd5dea99fe" (UID: "7044fd5d-1dbf-4337-9189-e1bd5dea99fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:48:20.337569 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.337536    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8969q\" (UniqueName: \"kubernetes.io/projected/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-kube-api-access-8969q\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:20.337569 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.337566    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:20.337569 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.337575    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7044fd5d-1dbf-4337-9189-e1bd5dea99fe-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:20.725498 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.725459    2574 generic.go:356] "Generic (PLEG): container finished" podID="7044fd5d-1dbf-4337-9189-e1bd5dea99fe" containerID="62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49" exitCode=0
Jul 24 00:48:20.725498 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.725501    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"
Jul 24 00:48:20.725712 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.725551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr" event={"ID":"7044fd5d-1dbf-4337-9189-e1bd5dea99fe","Type":"ContainerDied","Data":"62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49"}
Jul 24 00:48:20.725712 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.725600    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr" event={"ID":"7044fd5d-1dbf-4337-9189-e1bd5dea99fe","Type":"ContainerDied","Data":"7be1a3a97e20e290bbc089cb845d540f05316dfb0de85e61d5a66024fd302fe7"}
Jul 24 00:48:20.725712 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.725623    2574 scope.go:117] "RemoveContainer" containerID="62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49"
Jul 24 00:48:20.734157 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.734140    2574 scope.go:117] "RemoveContainer" containerID="62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49"
Jul 24 00:48:20.734411 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:48:20.734393    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49\": container with ID starting with 62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49 not found: ID does not exist" containerID="62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49"
Jul 24 00:48:20.734471 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.734421    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49"} err="failed to get container status \"62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49\": rpc error: code = NotFound desc = could not find container \"62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49\": container with ID starting with 62279f67e1ab92c027461d89213b5796303808826cbc6cf677bf3b63a0041b49 not found: ID does not exist"
Jul 24 00:48:20.747528 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.747506    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"]
Jul 24 00:48:20.750861 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:20.750839    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c28612-5dd74c77db-s2kqr"]
Jul 24 00:48:21.752307 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:21.752270    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7044fd5d-1dbf-4337-9189-e1bd5dea99fe" path="/var/lib/kubelet/pods/7044fd5d-1dbf-4337-9189-e1bd5dea99fe/volumes"
Jul 24 00:48:23.209435 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.209389    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5" podUID="56b8004d-9a8d-4e99-bacb-7435b1ffb80b" containerName="main" containerID="cri-o://bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3" gracePeriod=45
Jul 24 00:48:23.464402 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.464342    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:48:23.564980 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.564949    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-model-cache\") pod \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") "
Jul 24 00:48:23.565120 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.565002    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tmp-dir\") pod \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") "
Jul 24 00:48:23.565120 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.565048    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r966g\" (UniqueName: \"kubernetes.io/projected/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-kube-api-access-r966g\") pod \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") "
Jul 24 00:48:23.565120 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.565076    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-home\") pod \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") "
Jul 24 00:48:23.565240 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.565159    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tls-certs\") pod \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") "
Jul 24 00:48:23.565240 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.565186    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-dshm\") pod \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\" (UID: \"56b8004d-9a8d-4e99-bacb-7435b1ffb80b\") "
Jul 24 00:48:23.565307 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.565230    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-model-cache" (OuterVolumeSpecName: "model-cache") pod "56b8004d-9a8d-4e99-bacb-7435b1ffb80b" (UID: "56b8004d-9a8d-4e99-bacb-7435b1ffb80b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:23.565379 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.565309    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56b8004d-9a8d-4e99-bacb-7435b1ffb80b" (UID: "56b8004d-9a8d-4e99-bacb-7435b1ffb80b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:23.565451 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.565421    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-home" (OuterVolumeSpecName: "home") pod "56b8004d-9a8d-4e99-bacb-7435b1ffb80b" (UID: "56b8004d-9a8d-4e99-bacb-7435b1ffb80b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:23.565583 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.565546    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:23.565583 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.565562    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:23.565583 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.565575    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:23.567285 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.567257    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-dshm" (OuterVolumeSpecName: "dshm") pod "56b8004d-9a8d-4e99-bacb-7435b1ffb80b" (UID: "56b8004d-9a8d-4e99-bacb-7435b1ffb80b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:23.567285 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.567262    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56b8004d-9a8d-4e99-bacb-7435b1ffb80b" (UID: "56b8004d-9a8d-4e99-bacb-7435b1ffb80b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:48:23.567476 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.567326    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-kube-api-access-r966g" (OuterVolumeSpecName: "kube-api-access-r966g") pod "56b8004d-9a8d-4e99-bacb-7435b1ffb80b" (UID: "56b8004d-9a8d-4e99-bacb-7435b1ffb80b"). InnerVolumeSpecName "kube-api-access-r966g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:48:23.665985 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.665952    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r966g\" (UniqueName: \"kubernetes.io/projected/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-kube-api-access-r966g\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:23.665985 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.665979    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:23.665985 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.665990    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b8004d-9a8d-4e99-bacb-7435b1ffb80b-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:23.740489 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.740404    2574 generic.go:356] "Generic (PLEG): container finished" podID="56b8004d-9a8d-4e99-bacb-7435b1ffb80b" containerID="bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3" exitCode=0
Jul 24 00:48:23.740489 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.740446    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"
Jul 24 00:48:23.740661 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.740490    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5" event={"ID":"56b8004d-9a8d-4e99-bacb-7435b1ffb80b","Type":"ContainerDied","Data":"bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3"}
Jul 24 00:48:23.740661 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.740532    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5" event={"ID":"56b8004d-9a8d-4e99-bacb-7435b1ffb80b","Type":"ContainerDied","Data":"8ee3c7ce324af62766773900dd55775c818f3335448f62621f37c9d2b28628d2"}
Jul 24 00:48:23.740661 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.740551    2574 scope.go:117] "RemoveContainer" containerID="bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3"
Jul 24 00:48:23.749212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.749193    2574 scope.go:117] "RemoveContainer" containerID="bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3"
Jul 24 00:48:23.749479 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:48:23.749461    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3\": container with ID starting with bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3 not found: ID does not exist" containerID="bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3"
Jul 24 00:48:23.749601 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.749490    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3"} err="failed to get container status \"bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3\": rpc error: code = NotFound desc = could not find container \"bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3\": container with ID starting with bf9fb881447ffe9af019bceed4b3199d4cc20ed4409e433b06ee07d2add617b3 not found: ID does not exist"
Jul 24 00:48:23.775060 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.775033    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"]
Jul 24 00:48:23.778554 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:23.778532    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c28612/test-model-c28612-kserve-7457bf65fc-s64k5"]
Jul 24 00:48:24.160284 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.160215    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64" podUID="a3338c9f-0148-48ad-83eb-d51205641a21" containerName="main" containerID="cri-o://7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c" gracePeriod=45
Jul 24 00:48:24.411614 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.411554    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:48:24.574079 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.574040    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6xxk\" (UniqueName: \"kubernetes.io/projected/a3338c9f-0148-48ad-83eb-d51205641a21-kube-api-access-j6xxk\") pod \"a3338c9f-0148-48ad-83eb-d51205641a21\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") "
Jul 24 00:48:24.574272 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.574109    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-home\") pod \"a3338c9f-0148-48ad-83eb-d51205641a21\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") "
Jul 24 00:48:24.574272 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.574130    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-model-cache\") pod \"a3338c9f-0148-48ad-83eb-d51205641a21\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") "
Jul 24 00:48:24.574272 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.574153    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-dshm\") pod \"a3338c9f-0148-48ad-83eb-d51205641a21\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") "
Jul 24 00:48:24.574272 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.574220    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-tmp-dir\") pod \"a3338c9f-0148-48ad-83eb-d51205641a21\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") "
Jul 24 00:48:24.574272 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.574249    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3338c9f-0148-48ad-83eb-d51205641a21-tls-certs\") pod \"a3338c9f-0148-48ad-83eb-d51205641a21\" (UID: \"a3338c9f-0148-48ad-83eb-d51205641a21\") "
Jul 24 00:48:24.574567 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.574327    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-home" (OuterVolumeSpecName: "home") pod "a3338c9f-0148-48ad-83eb-d51205641a21" (UID: "a3338c9f-0148-48ad-83eb-d51205641a21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:24.574567 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.574424    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3338c9f-0148-48ad-83eb-d51205641a21" (UID: "a3338c9f-0148-48ad-83eb-d51205641a21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:24.574567 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.574541    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3338c9f-0148-48ad-83eb-d51205641a21" (UID: "a3338c9f-0148-48ad-83eb-d51205641a21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:24.574567 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.574559    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:24.574723 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.574578    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:24.576327 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.576295    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3338c9f-0148-48ad-83eb-d51205641a21-kube-api-access-j6xxk" (OuterVolumeSpecName: "kube-api-access-j6xxk") pod "a3338c9f-0148-48ad-83eb-d51205641a21" (UID: "a3338c9f-0148-48ad-83eb-d51205641a21"). InnerVolumeSpecName "kube-api-access-j6xxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:48:24.576413 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.576357    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3338c9f-0148-48ad-83eb-d51205641a21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3338c9f-0148-48ad-83eb-d51205641a21" (UID: "a3338c9f-0148-48ad-83eb-d51205641a21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:48:24.576413 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.576386    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-dshm" (OuterVolumeSpecName: "dshm") pod "a3338c9f-0148-48ad-83eb-d51205641a21" (UID: "a3338c9f-0148-48ad-83eb-d51205641a21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:24.675034 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.674956    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:24.675034 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.674983    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3338c9f-0148-48ad-83eb-d51205641a21-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:24.675034 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.674998    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6xxk\" (UniqueName: \"kubernetes.io/projected/a3338c9f-0148-48ad-83eb-d51205641a21-kube-api-access-j6xxk\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:24.675034 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.675010    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3338c9f-0148-48ad-83eb-d51205641a21-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:24.746477 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.746448    2574 generic.go:356] "Generic (PLEG): container finished" podID="a3338c9f-0148-48ad-83eb-d51205641a21" containerID="7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c" exitCode=0
Jul 24 00:48:24.746631 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.746482    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64" event={"ID":"a3338c9f-0148-48ad-83eb-d51205641a21","Type":"ContainerDied","Data":"7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c"}
Jul 24 00:48:24.746631 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.746494    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"
Jul 24 00:48:24.746631 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.746505    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64" event={"ID":"a3338c9f-0148-48ad-83eb-d51205641a21","Type":"ContainerDied","Data":"3e98376b0fd2e1db2e367d883dfcd65cdca7d1fc55fe3d6262f0c06c089f6149"}
Jul 24 00:48:24.746631 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.746520    2574 scope.go:117] "RemoveContainer" containerID="7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c"
Jul 24 00:48:24.755486 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.755452    2574 scope.go:117] "RemoveContainer" containerID="7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c"
Jul 24 00:48:24.755782 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:48:24.755763    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c\": container with ID starting with 7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c not found: ID does not exist" containerID="7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c"
Jul 24 00:48:24.755856 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.755791    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c"} err="failed to get container status \"7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c\": rpc error: code = NotFound desc = could not find container \"7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c\": container with ID starting with 7f9d596db551c27d0d3d67baf7944ef03449d76b5d453e6ab21b242076e08e2c not found: ID does not exist"
Jul 24 00:48:24.778256 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.778232    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"]
Jul 24 00:48:24.784588 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:24.784564    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5f040e/test-model-5f040e-kserve-584488bbb5-fss64"]
Jul 24 00:48:25.752213 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:25.752180    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56b8004d-9a8d-4e99-bacb-7435b1ffb80b" path="/var/lib/kubelet/pods/56b8004d-9a8d-4e99-bacb-7435b1ffb80b/volumes"
Jul 24 00:48:25.752598 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:25.752574    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3338c9f-0148-48ad-83eb-d51205641a21" path="/var/lib/kubelet/pods/a3338c9f-0148-48ad-83eb-d51205641a21/volumes"
Jul 24 00:48:31.110326 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:31.110279    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"]
Jul 24 00:48:31.110710 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:31.110506    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerName="istio-proxy" containerID="cri-o://e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5" gracePeriod=30
Jul 24 00:48:36.364475 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.364448    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:48:36.470644 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.470556    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-envoy\") pod \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") "
Jul 24 00:48:36.470644 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.470595    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-credential-socket\") pod \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") "
Jul 24 00:48:36.470644 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.470619    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-token\") pod \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") "
Jul 24 00:48:36.470927 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.470650    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-podinfo\") pod \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") "
Jul 24 00:48:36.470927 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.470671    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcbfb\" (UniqueName: \"kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-kube-api-access-wcbfb\") pod \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") "
Jul 24 00:48:36.470927 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.470685    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-data\") pod \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") "
Jul 24 00:48:36.470927 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.470700    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-socket\") pod \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") "
Jul 24 00:48:36.470927 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.470738    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-certs\") pod \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") "
Jul 24 00:48:36.470927 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.470766    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istiod-ca-cert\") pod \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\" (UID: \"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1\") "
Jul 24 00:48:36.471246 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.470915    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" (UID: "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:36.471246 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.471068    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-credential-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:36.471246 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.471181    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" (UID: "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:36.471449 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.471278    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" (UID: "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:36.471449 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.471334    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" (UID: "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:48:36.471571 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.471544    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" (UID: "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:36.472868 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.472838    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" (UID: "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:48:36.472986 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.472873    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" (UID: "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 00:48:36.473233 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.473209    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" (UID: "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:48:36.473291 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.473248    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-kube-api-access-wcbfb" (OuterVolumeSpecName: "kube-api-access-wcbfb") pod "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" (UID: "f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1"). InnerVolumeSpecName "kube-api-access-wcbfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:48:36.571487 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.571452    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:36.571487 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.571477    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-workload-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:36.571487 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.571489    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istiod-ca-cert\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:36.571707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.571500    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-envoy\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:36.571707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.571511    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-token\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:36.571707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.571522    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-podinfo\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:36.571707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.571533    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcbfb\" (UniqueName: \"kubernetes.io/projected/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-kube-api-access-wcbfb\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:36.571707 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.571544    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1-istio-data\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:36.798114 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.798030    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerID="e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5" exitCode=0
Jul 24 00:48:36.798114 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.798078    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"
Jul 24 00:48:36.798359 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.798114    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" event={"ID":"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1","Type":"ContainerDied","Data":"e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5"}
Jul 24 00:48:36.798359 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.798179    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9" event={"ID":"f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1","Type":"ContainerDied","Data":"0f2750429d43faef9f413f940409f6248b84d25becea547188378457f46d5332"}
Jul 24 00:48:36.798359 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.798209    2574 scope.go:117] "RemoveContainer" containerID="e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5"
Jul 24 00:48:36.807270 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.807253    2574 scope.go:117] "RemoveContainer" containerID="65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252"
Jul 24 00:48:36.814607 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.814582    2574 scope.go:117] "RemoveContainer" containerID="e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5"
Jul 24 00:48:36.814843 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:48:36.814825    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5\": container with ID starting with e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5 not found: ID does not exist" containerID="e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5"
Jul 24 00:48:36.814928 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.814855    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5"} err="failed to get container status \"e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5\": rpc error: code = NotFound desc = could not find container \"e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5\": container with ID starting with e8055f4a826540af3a58235c823f56b64df839b81c40084f531f5b1ec18a27b5 not found: ID does not exist"
Jul 24 00:48:36.814928 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.814878    2574 scope.go:117] "RemoveContainer" containerID="65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252"
Jul 24 00:48:36.815149 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:48:36.815132    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252\": container with ID starting with 65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252 not found: ID does not exist" containerID="65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252"
Jul 24 00:48:36.815186 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.815155    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252"} err="failed to get container status \"65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252\": rpc error: code = NotFound desc = could not find container \"65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252\": container with ID starting with 65e6f849c9e4676fc0b56bbf5512615dc2ffa16c56db95bea2ac4e0f778c3252 not found: ID does not exist"
Jul 24 00:48:36.819328 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.819291    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"]
Jul 24 00:48:36.823890 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:36.823872    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c28612-openshift-default-66bc8d44cd-hcsr9"]
Jul 24 00:48:37.752347 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:37.752296    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" path="/var/lib/kubelet/pods/f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1/volumes"
Jul 24 00:48:42.077385 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.077352    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"]
Jul 24 00:48:42.077871 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.077606    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp" podUID="8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2" containerName="maas-api" containerID="cri-o://d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d" gracePeriod=30
Jul 24 00:48:42.319921 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.319898    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:48:42.417846 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.417772    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fblsm\" (UniqueName: \"kubernetes.io/projected/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-kube-api-access-fblsm\") pod \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") "
Jul 24 00:48:42.417846 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.417835    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-openshift-service-ca\") pod \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") "
Jul 24 00:48:42.418018 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.417881    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-maas-api-tls\") pod \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\" (UID: \"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2\") "
Jul 24 00:48:42.418221 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.418198    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2" (UID: "8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:48:42.419927 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.419905    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2" (UID: "8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:48:42.420027 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.420009    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-kube-api-access-fblsm" (OuterVolumeSpecName: "kube-api-access-fblsm") pod "8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2" (UID: "8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2"). InnerVolumeSpecName "kube-api-access-fblsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:48:42.519096 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.519066    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:42.519096 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.519093    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:42.519265 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.519104    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fblsm\" (UniqueName: \"kubernetes.io/projected/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2-kube-api-access-fblsm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:48:42.821847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.821814    2574 generic.go:356] "Generic (PLEG): container finished" podID="8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2" containerID="d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d" exitCode=0
Jul 24 00:48:42.822006 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.821865    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"
Jul 24 00:48:42.822006 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.821865    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp" event={"ID":"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2","Type":"ContainerDied","Data":"d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d"}
Jul 24 00:48:42.822006 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.821971    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp" event={"ID":"8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2","Type":"ContainerDied","Data":"0268e3d6d0fce8236e7f79e3571ad57e7c8223c50c6970c8d1b6904516c4bbd9"}
Jul 24 00:48:42.822006 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.821997    2574 scope.go:117] "RemoveContainer" containerID="d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d"
Jul 24 00:48:42.830163 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.830145    2574 scope.go:117] "RemoveContainer" containerID="d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d"
Jul 24 00:48:42.830434 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:48:42.830406    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d\": container with ID starting with d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d not found: ID does not exist" containerID="d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d"
Jul 24 00:48:42.830523 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.830439    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d"} err="failed to get container status \"d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d\": rpc error: code = NotFound desc = could not find container \"d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d\": container with ID starting with d8e26e720bc77ab50bf339b325d76e89cd6e10b9be81bd0b61eba8417044106d not found: ID does not exist"
Jul 24 00:48:42.842158 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.842133    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"]
Jul 24 00:48:42.848550 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:42.848528    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f040e-568fdfbbd9-xrltp"]
Jul 24 00:48:43.751798 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:43.751756    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2" path="/var/lib/kubelet/pods/8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2/volumes"
Jul 24 00:48:53.949410 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.949368    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"]
Jul 24 00:48:53.950025 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.949999    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerName="istio-proxy"
Jul 24 00:48:53.950132 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950029    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerName="istio-proxy"
Jul 24 00:48:53.950132 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950050    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b8004d-9a8d-4e99-bacb-7435b1ffb80b" containerName="main"
Jul 24 00:48:53.950132 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950065    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b8004d-9a8d-4e99-bacb-7435b1ffb80b" containerName="main"
Jul 24 00:48:53.950132 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950097    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2" containerName="maas-api"
Jul 24 00:48:53.950132 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950107    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2" containerName="maas-api"
Jul 24 00:48:53.950132 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950117    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7044fd5d-1dbf-4337-9189-e1bd5dea99fe" containerName="maas-api"
Jul 24 00:48:53.950132 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950126    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7044fd5d-1dbf-4337-9189-e1bd5dea99fe" containerName="maas-api"
Jul 24 00:48:53.950494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950138    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3338c9f-0148-48ad-83eb-d51205641a21" containerName="main"
Jul 24 00:48:53.950494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950146    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3338c9f-0148-48ad-83eb-d51205641a21" containerName="main"
Jul 24 00:48:53.950494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950297    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerName="istio-proxy"
Jul 24 00:48:53.950494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950336    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7044fd5d-1dbf-4337-9189-e1bd5dea99fe" containerName="maas-api"
Jul 24 00:48:53.950494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950348    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3338c9f-0148-48ad-83eb-d51205641a21" containerName="main"
Jul 24 00:48:53.950494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950365    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a76dc31-c4a9-433d-9fbe-1f6cb45b4bc2" containerName="maas-api"
Jul 24 00:48:53.950494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.950374    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b8004d-9a8d-4e99-bacb-7435b1ffb80b" containerName="main"
Jul 24 00:48:53.954183 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.954149    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:53.956897 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.956878    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-796b6a-openshift-default-dockercfg-5tl2x\""
Jul 24 00:48:53.964919 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:53.964897    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"]
Jul 24 00:48:54.016838 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.016815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-data\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.016838 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.016843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-token\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.016999 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.016876    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-certs\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.016999 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.016892    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-podinfo\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.017088 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.016989    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-credential-socket\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.017088 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.017024    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-socket\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.017088 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.017047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45866176-c1f4-47d2-a2c0-ceaa41379540-istiod-ca-cert\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.017245 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.017111    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w646\" (UniqueName: \"kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-kube-api-access-4w646\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.017245 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.017165    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-envoy\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.118201 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.118149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-certs\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.118201 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.118206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-podinfo\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.118494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.118242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-credential-socket\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.118494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.118272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-socket\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.118494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.118296    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45866176-c1f4-47d2-a2c0-ceaa41379540-istiod-ca-cert\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.118494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.118355    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w646\" (UniqueName: \"kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-kube-api-access-4w646\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.118494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.118394    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-envoy\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.118494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.118449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-data\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.118494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.118474    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-token\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.118842 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.118655    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-certs\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.118842 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.118661    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-socket\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.119019 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.119000    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-credential-socket\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.119189 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.119170    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-data\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.119361 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.119338    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45866176-c1f4-47d2-a2c0-ceaa41379540-istiod-ca-cert\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.120834 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.120805    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-envoy\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.120963 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.120943    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-podinfo\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.126546 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.126519    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-token\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.126651 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.126563    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w646\" (UniqueName: \"kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-kube-api-access-4w646\") pod \"e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") " pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.264782 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.264688    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:54.400871 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.400845    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"]
Jul 24 00:48:54.403354 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:48:54.403304    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45866176_c1f4_47d2_a2c0_ceaa41379540.slice/crio-26f88118718535eb19e6284e444f1c1502dfdf4ffa992eb02cf1d09b205f39a4 WatchSource:0}: Error finding container 26f88118718535eb19e6284e444f1c1502dfdf4ffa992eb02cf1d09b205f39a4: Status 404 returned error can't find the container with id 26f88118718535eb19e6284e444f1c1502dfdf4ffa992eb02cf1d09b205f39a4
Jul 24 00:48:54.405424 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.405395    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:48:54.405499 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.405456    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:48:54.405499 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.405480    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:48:54.873563 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.873533    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" event={"ID":"45866176-c1f4-47d2-a2c0-ceaa41379540","Type":"ContainerStarted","Data":"5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96"}
Jul 24 00:48:54.873708 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.873583    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" event={"ID":"45866176-c1f4-47d2-a2c0-ceaa41379540","Type":"ContainerStarted","Data":"26f88118718535eb19e6284e444f1c1502dfdf4ffa992eb02cf1d09b205f39a4"}
Jul 24 00:48:54.892996 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:54.892950    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" podStartSLOduration=1.892934329 podStartE2EDuration="1.892934329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:54.892035111 +0000 UTC m=+2297.740353854" watchObservedRunningTime="2026-07-24 00:48:54.892934329 +0000 UTC m=+2297.741253071"
Jul 24 00:48:55.265580 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:55.265494    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:55.270415 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:55.270392    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:55.877887 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:55.877855    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:55.878842 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:55.878823    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:48:56.926375 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:56.926343    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"]
Jul 24 00:48:56.926769 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:56.926681    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerName="istio-proxy"
Jul 24 00:48:56.926769 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:56.926697    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerName="istio-proxy"
Jul 24 00:48:56.926873 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:56.926787    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a8aff7-c16d-4374-8ebe-8bc455ee3aa1" containerName="istio-proxy"
Jul 24 00:48:56.929684 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:56.929665    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:56.932247 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:56.932222    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-796b6a-serving-cert\""
Jul 24 00:48:56.932366 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:56.932353    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nv7dj\""
Jul 24 00:48:56.933342 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:56.933302    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 00:48:56.933435 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:56.933338    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 00:48:56.942657 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:56.942633    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"]
Jul 24 00:48:57.043475 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.043440    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1623325-a609-479b-9b97-7000897d7eaa-openshift-service-ca\") pod \"maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:57.043655 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.043492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1623325-a609-479b-9b97-7000897d7eaa-maas-api-tls\") pod \"maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:57.043655 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.043522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgbg\" (UniqueName: \"kubernetes.io/projected/b1623325-a609-479b-9b97-7000897d7eaa-kube-api-access-9vgbg\") pod \"maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:57.144598 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.144570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1623325-a609-479b-9b97-7000897d7eaa-openshift-service-ca\") pod \"maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:57.144787 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.144614    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1623325-a609-479b-9b97-7000897d7eaa-maas-api-tls\") pod \"maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:57.144787 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.144636    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgbg\" (UniqueName: \"kubernetes.io/projected/b1623325-a609-479b-9b97-7000897d7eaa-kube-api-access-9vgbg\") pod \"maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:57.144787 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:48:57.144741    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-serving-cert: secret "maas-api-e2e-api-a-796b6a-serving-cert" not found
Jul 24 00:48:57.144967 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:48:57.144827    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1623325-a609-479b-9b97-7000897d7eaa-maas-api-tls podName:b1623325-a609-479b-9b97-7000897d7eaa nodeName:}" failed. No retries permitted until 2026-07-24 00:48:57.644805679 +0000 UTC m=+2300.493124416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1623325-a609-479b-9b97-7000897d7eaa-maas-api-tls") pod "maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v" (UID: "b1623325-a609-479b-9b97-7000897d7eaa") : secret "maas-api-e2e-api-a-796b6a-serving-cert" not found
Jul 24 00:48:57.145372 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.145351    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1623325-a609-479b-9b97-7000897d7eaa-openshift-service-ca\") pod \"maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:57.155690 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.155668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgbg\" (UniqueName: \"kubernetes.io/projected/b1623325-a609-479b-9b97-7000897d7eaa-kube-api-access-9vgbg\") pod \"maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:57.649777 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.649747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1623325-a609-479b-9b97-7000897d7eaa-maas-api-tls\") pod \"maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:57.652179 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.652156    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1623325-a609-479b-9b97-7000897d7eaa-maas-api-tls\") pod \"maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:57.840787 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.840754    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:48:57.974409 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.974356    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"]
Jul 24 00:48:57.977259 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:48:57.977217    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1623325_a609_479b_9b97_7000897d7eaa.slice/crio-ea4d3cee79d8b9b655dfb381ef5f47ff74a1ad41887b45698d205c6a83be9cfc WatchSource:0}: Error finding container ea4d3cee79d8b9b655dfb381ef5f47ff74a1ad41887b45698d205c6a83be9cfc: Status 404 returned error can't find the container with id ea4d3cee79d8b9b655dfb381ef5f47ff74a1ad41887b45698d205c6a83be9cfc
Jul 24 00:48:57.978743 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:57.978722    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:48:58.890954 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:58.890920    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v" event={"ID":"b1623325-a609-479b-9b97-7000897d7eaa","Type":"ContainerStarted","Data":"3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867"}
Jul 24 00:48:58.891147 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:58.890969    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v" event={"ID":"b1623325-a609-479b-9b97-7000897d7eaa","Type":"ContainerStarted","Data":"ea4d3cee79d8b9b655dfb381ef5f47ff74a1ad41887b45698d205c6a83be9cfc"}
Jul 24 00:48:58.891147 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:48:58.891010    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:49:04.902634 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:04.902554    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:49:04.919146 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:04.919086    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v" podStartSLOduration=8.500369972 podStartE2EDuration="8.919067342s" totalImagesPullingTime="418.69737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:58.919453661 +0000 UTC m=+2301.767772403" watchObservedRunningTime="2026-07-24 00:49:04.919067342 +0000 UTC m=+2307.767386085"
Jul 24 00:49:05.074710 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.074671    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"]
Jul 24 00:49:05.078719 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.078694    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:05.081640 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.081611    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a4c256-serving-cert\""
Jul 24 00:49:05.091720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.091610    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"]
Jul 24 00:49:05.215232 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.215139    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaaac91d-d17b-4aaa-852b-809a780ed638-maas-api-tls\") pod \"maas-api-e2e-api-b-a4c256-6f6c879f95-722np\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:05.215232 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.215190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaaac91d-d17b-4aaa-852b-809a780ed638-openshift-service-ca\") pod \"maas-api-e2e-api-b-a4c256-6f6c879f95-722np\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:05.215232 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.215233    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4p89\" (UniqueName: \"kubernetes.io/projected/aaaac91d-d17b-4aaa-852b-809a780ed638-kube-api-access-t4p89\") pod \"maas-api-e2e-api-b-a4c256-6f6c879f95-722np\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:05.316632 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.316598    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaaac91d-d17b-4aaa-852b-809a780ed638-maas-api-tls\") pod \"maas-api-e2e-api-b-a4c256-6f6c879f95-722np\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:05.316812 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.316643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaaac91d-d17b-4aaa-852b-809a780ed638-openshift-service-ca\") pod \"maas-api-e2e-api-b-a4c256-6f6c879f95-722np\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:05.316812 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.316679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4p89\" (UniqueName: \"kubernetes.io/projected/aaaac91d-d17b-4aaa-852b-809a780ed638-kube-api-access-t4p89\") pod \"maas-api-e2e-api-b-a4c256-6f6c879f95-722np\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:05.316812 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:49:05.316741    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-serving-cert: secret "maas-api-e2e-api-b-a4c256-serving-cert" not found
Jul 24 00:49:05.316812 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:49:05.316813    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaaac91d-d17b-4aaa-852b-809a780ed638-maas-api-tls podName:aaaac91d-d17b-4aaa-852b-809a780ed638 nodeName:}" failed. No retries permitted until 2026-07-24 00:49:05.816789164 +0000 UTC m=+2308.665107883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aaaac91d-d17b-4aaa-852b-809a780ed638-maas-api-tls") pod "maas-api-e2e-api-b-a4c256-6f6c879f95-722np" (UID: "aaaac91d-d17b-4aaa-852b-809a780ed638") : secret "maas-api-e2e-api-b-a4c256-serving-cert" not found
Jul 24 00:49:05.317571 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.317544    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaaac91d-d17b-4aaa-852b-809a780ed638-openshift-service-ca\") pod \"maas-api-e2e-api-b-a4c256-6f6c879f95-722np\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:05.334189 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.334161    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4p89\" (UniqueName: \"kubernetes.io/projected/aaaac91d-d17b-4aaa-852b-809a780ed638-kube-api-access-t4p89\") pod \"maas-api-e2e-api-b-a4c256-6f6c879f95-722np\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:05.821892 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.821859    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaaac91d-d17b-4aaa-852b-809a780ed638-maas-api-tls\") pod \"maas-api-e2e-api-b-a4c256-6f6c879f95-722np\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:05.824272 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.824247    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaaac91d-d17b-4aaa-852b-809a780ed638-maas-api-tls\") pod \"maas-api-e2e-api-b-a4c256-6f6c879f95-722np\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:05.993124 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:05.993086    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:06.134016 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:06.133980    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"]
Jul 24 00:49:06.137863 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:49:06.137834    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaaac91d_d17b_4aaa_852b_809a780ed638.slice/crio-e00cc27623384524bc81183310372f6453c0bfe361baa0a7459dd1d8bc76ccb7 WatchSource:0}: Error finding container e00cc27623384524bc81183310372f6453c0bfe361baa0a7459dd1d8bc76ccb7: Status 404 returned error can't find the container with id e00cc27623384524bc81183310372f6453c0bfe361baa0a7459dd1d8bc76ccb7
Jul 24 00:49:06.937033 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:06.936998    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np" event={"ID":"aaaac91d-d17b-4aaa-852b-809a780ed638","Type":"ContainerStarted","Data":"9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4"}
Jul 24 00:49:06.937198 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:06.937057    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np" event={"ID":"aaaac91d-d17b-4aaa-852b-809a780ed638","Type":"ContainerStarted","Data":"e00cc27623384524bc81183310372f6453c0bfe361baa0a7459dd1d8bc76ccb7"}
Jul 24 00:49:06.937198 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:06.937144    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:06.962290 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:06.962237    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np" podStartSLOduration=1.667299227 podStartE2EDuration="1.962222298s" totalImagesPullingTime="294.923071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:49:06.960674286 +0000 UTC m=+2309.808993027" watchObservedRunningTime="2026-07-24 00:49:06.962222298 +0000 UTC m=+2309.810541039"
Jul 24 00:49:07.942941 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:07.942912    2574 generic.go:356] "Generic (PLEG): container finished" podID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerID="5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96" exitCode=0
Jul 24 00:49:07.943512 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:07.943008    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" event={"ID":"45866176-c1f4-47d2-a2c0-ceaa41379540","Type":"ContainerDied","Data":"5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96"}
Jul 24 00:49:07.943512 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:07.943427    2574 scope.go:117] "RemoveContainer" containerID="5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96"
Jul 24 00:49:07.944400 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:07.944375    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:49:07.944508 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:07.944453    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:49:07.944573 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:07.944532    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:49:08.952594 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:08.952556    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" event={"ID":"45866176-c1f4-47d2-a2c0-ceaa41379540","Type":"ContainerStarted","Data":"d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5"}
Jul 24 00:49:09.265490 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:09.265395    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:49:09.266664 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:09.266631    2574 patch_prober.go:28] interesting pod/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 24 00:49:09.266806 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:09.266691    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 24 00:49:10.266036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:10.265997    2574 patch_prober.go:28] interesting pod/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 24 00:49:10.266488 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:10.266058    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 24 00:49:10.878728 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:10.878698    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:49:11.265503 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:11.265413    2574 patch_prober.go:28] interesting pod/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 24 00:49:11.265503 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:11.265470    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 24 00:49:12.265422 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:12.265376    2574 patch_prober.go:28] interesting pod/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 24 00:49:12.265990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:12.265438    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 24 00:49:12.949196 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:12.949168    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:49:13.269754 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:13.269673    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:49:13.270771 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:13.270749    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:49:33.407723 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.407646    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"]
Jul 24 00:49:33.408133 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.407861    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v" podUID="b1623325-a609-479b-9b97-7000897d7eaa" containerName="maas-api" containerID="cri-o://3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867" gracePeriod=30
Jul 24 00:49:33.656493 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.656467    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:49:33.765716 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.765639    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1623325-a609-479b-9b97-7000897d7eaa-openshift-service-ca\") pod \"b1623325-a609-479b-9b97-7000897d7eaa\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") "
Jul 24 00:49:33.765716 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.765691    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1623325-a609-479b-9b97-7000897d7eaa-maas-api-tls\") pod \"b1623325-a609-479b-9b97-7000897d7eaa\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") "
Jul 24 00:49:33.765943 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.765719    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vgbg\" (UniqueName: \"kubernetes.io/projected/b1623325-a609-479b-9b97-7000897d7eaa-kube-api-access-9vgbg\") pod \"b1623325-a609-479b-9b97-7000897d7eaa\" (UID: \"b1623325-a609-479b-9b97-7000897d7eaa\") "
Jul 24 00:49:33.766037 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.766011    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1623325-a609-479b-9b97-7000897d7eaa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1623325-a609-479b-9b97-7000897d7eaa" (UID: "b1623325-a609-479b-9b97-7000897d7eaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:49:33.767811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.767783    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1623325-a609-479b-9b97-7000897d7eaa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1623325-a609-479b-9b97-7000897d7eaa" (UID: "b1623325-a609-479b-9b97-7000897d7eaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:49:33.767811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.767784    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1623325-a609-479b-9b97-7000897d7eaa-kube-api-access-9vgbg" (OuterVolumeSpecName: "kube-api-access-9vgbg") pod "b1623325-a609-479b-9b97-7000897d7eaa" (UID: "b1623325-a609-479b-9b97-7000897d7eaa"). InnerVolumeSpecName "kube-api-access-9vgbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:49:33.866369 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.866333    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1623325-a609-479b-9b97-7000897d7eaa-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:33.866369 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.866359    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1623325-a609-479b-9b97-7000897d7eaa-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:33.866369 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:33.866369    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgbg\" (UniqueName: \"kubernetes.io/projected/b1623325-a609-479b-9b97-7000897d7eaa-kube-api-access-9vgbg\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:34.063800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:34.063722    2574 generic.go:356] "Generic (PLEG): container finished" podID="b1623325-a609-479b-9b97-7000897d7eaa" containerID="3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867" exitCode=0
Jul 24 00:49:34.063800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:34.063769    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"
Jul 24 00:49:34.063800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:34.063791    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v" event={"ID":"b1623325-a609-479b-9b97-7000897d7eaa","Type":"ContainerDied","Data":"3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867"}
Jul 24 00:49:34.064010 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:34.063831    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v" event={"ID":"b1623325-a609-479b-9b97-7000897d7eaa","Type":"ContainerDied","Data":"ea4d3cee79d8b9b655dfb381ef5f47ff74a1ad41887b45698d205c6a83be9cfc"}
Jul 24 00:49:34.064010 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:34.063854    2574 scope.go:117] "RemoveContainer" containerID="3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867"
Jul 24 00:49:34.072518 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:34.072503    2574 scope.go:117] "RemoveContainer" containerID="3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867"
Jul 24 00:49:34.072749 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:49:34.072733    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867\": container with ID starting with 3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867 not found: ID does not exist" containerID="3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867"
Jul 24 00:49:34.072801 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:34.072757    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867"} err="failed to get container status \"3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867\": rpc error: code = NotFound desc = could not find container \"3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867\": container with ID starting with 3b0a5c960114b5802d93185f5a05c98cefe1ae119710c4cf40d895372940f867 not found: ID does not exist"
Jul 24 00:49:34.084775 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:34.084749    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"]
Jul 24 00:49:34.088394 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:34.088375    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-796b6a-7668b4fdc8-crn8v"]
Jul 24 00:49:35.752046 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:35.752012    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1623325-a609-479b-9b97-7000897d7eaa" path="/var/lib/kubelet/pods/b1623325-a609-479b-9b97-7000897d7eaa/volumes"
Jul 24 00:49:44.679253 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:44.679216    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"]
Jul 24 00:49:44.679848 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:44.679450    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerName="istio-proxy" containerID="cri-o://d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5" gracePeriod=30
Jul 24 00:49:49.919153 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:49.919128    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:49:50.002214 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002142    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-envoy\") pod \"45866176-c1f4-47d2-a2c0-ceaa41379540\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") "
Jul 24 00:49:50.002214 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002180    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45866176-c1f4-47d2-a2c0-ceaa41379540-istiod-ca-cert\") pod \"45866176-c1f4-47d2-a2c0-ceaa41379540\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") "
Jul 24 00:49:50.002214 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002202    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-token\") pod \"45866176-c1f4-47d2-a2c0-ceaa41379540\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") "
Jul 24 00:49:50.002492 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002228    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-podinfo\") pod \"45866176-c1f4-47d2-a2c0-ceaa41379540\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") "
Jul 24 00:49:50.002492 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002267    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-certs\") pod \"45866176-c1f4-47d2-a2c0-ceaa41379540\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") "
Jul 24 00:49:50.002492 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002360    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-credential-socket\") pod \"45866176-c1f4-47d2-a2c0-ceaa41379540\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") "
Jul 24 00:49:50.002492 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002392    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-socket\") pod \"45866176-c1f4-47d2-a2c0-ceaa41379540\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") "
Jul 24 00:49:50.002492 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002425    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w646\" (UniqueName: \"kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-kube-api-access-4w646\") pod \"45866176-c1f4-47d2-a2c0-ceaa41379540\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") "
Jul 24 00:49:50.002492 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002462    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-data\") pod \"45866176-c1f4-47d2-a2c0-ceaa41379540\" (UID: \"45866176-c1f4-47d2-a2c0-ceaa41379540\") "
Jul 24 00:49:50.002798 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002561    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45866176-c1f4-47d2-a2c0-ceaa41379540" (UID: "45866176-c1f4-47d2-a2c0-ceaa41379540"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:49:50.002798 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002625    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45866176-c1f4-47d2-a2c0-ceaa41379540-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45866176-c1f4-47d2-a2c0-ceaa41379540" (UID: "45866176-c1f4-47d2-a2c0-ceaa41379540"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:49:50.002798 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002770    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45866176-c1f4-47d2-a2c0-ceaa41379540" (UID: "45866176-c1f4-47d2-a2c0-ceaa41379540"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:49:50.002798 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002788    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:50.003009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002809    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45866176-c1f4-47d2-a2c0-ceaa41379540-istiod-ca-cert\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:50.003009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.002833    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45866176-c1f4-47d2-a2c0-ceaa41379540" (UID: "45866176-c1f4-47d2-a2c0-ceaa41379540"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:49:50.003231 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.003205    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-data" (OuterVolumeSpecName: "istio-data") pod "45866176-c1f4-47d2-a2c0-ceaa41379540" (UID: "45866176-c1f4-47d2-a2c0-ceaa41379540"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:49:50.004441 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.004416    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45866176-c1f4-47d2-a2c0-ceaa41379540" (UID: "45866176-c1f4-47d2-a2c0-ceaa41379540"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:49:50.004520 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.004478    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-token" (OuterVolumeSpecName: "istio-token") pod "45866176-c1f4-47d2-a2c0-ceaa41379540" (UID: "45866176-c1f4-47d2-a2c0-ceaa41379540"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:49:50.004884 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.004861    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-kube-api-access-4w646" (OuterVolumeSpecName: "kube-api-access-4w646") pod "45866176-c1f4-47d2-a2c0-ceaa41379540" (UID: "45866176-c1f4-47d2-a2c0-ceaa41379540"). InnerVolumeSpecName "kube-api-access-4w646". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:49:50.004985 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.004877    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45866176-c1f4-47d2-a2c0-ceaa41379540" (UID: "45866176-c1f4-47d2-a2c0-ceaa41379540"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 00:49:50.103575 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.103538    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-credential-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:50.103575 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.103572    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-workload-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:50.103575 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.103582    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w646\" (UniqueName: \"kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-kube-api-access-4w646\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:50.103781 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.103592    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-data\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:50.103781 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.103600    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-envoy\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:50.103781 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.103607    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-token\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:50.103781 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.103615    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45866176-c1f4-47d2-a2c0-ceaa41379540-istio-podinfo\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:49:50.129881 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.129856    2574 generic.go:356] "Generic (PLEG): container finished" podID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerID="d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5" exitCode=0
Jul 24 00:49:50.130008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.129913    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"
Jul 24 00:49:50.130008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.129919    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" event={"ID":"45866176-c1f4-47d2-a2c0-ceaa41379540","Type":"ContainerDied","Data":"d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5"}
Jul 24 00:49:50.130109 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.130035    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq" event={"ID":"45866176-c1f4-47d2-a2c0-ceaa41379540","Type":"ContainerDied","Data":"26f88118718535eb19e6284e444f1c1502dfdf4ffa992eb02cf1d09b205f39a4"}
Jul 24 00:49:50.130109 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.130057    2574 scope.go:117] "RemoveContainer" containerID="d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5"
Jul 24 00:49:50.138499 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.138476    2574 scope.go:117] "RemoveContainer" containerID="5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96"
Jul 24 00:49:50.145492 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.145475    2574 scope.go:117] "RemoveContainer" containerID="d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5"
Jul 24 00:49:50.145713 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:49:50.145698    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5\": container with ID starting with d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5 not found: ID does not exist" containerID="d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5"
Jul 24 00:49:50.145751 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.145722    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5"} err="failed to get container status \"d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5\": rpc error: code = NotFound desc = could not find container \"d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5\": container with ID starting with d6bb2be4225851665947f9a3d3160acfb0a59df26267cdbaaafc87552dda61c5 not found: ID does not exist"
Jul 24 00:49:50.145751 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.145738    2574 scope.go:117] "RemoveContainer" containerID="5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96"
Jul 24 00:49:50.145961 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:49:50.145945    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96\": container with ID starting with 5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96 not found: ID does not exist" containerID="5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96"
Jul 24 00:49:50.146000 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.145968    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96"} err="failed to get container status \"5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96\": rpc error: code = NotFound desc = could not find container \"5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96\": container with ID starting with 5633b19513330f7874c723102558f8dd291e91447e99b2a9ad6a7616022d5e96 not found: ID does not exist"
Jul 24 00:49:50.150526 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.150504    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"]
Jul 24 00:49:50.155060 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:50.155015    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-796b6a-openshift-default-64f5dfd774-4s2kq"]
Jul 24 00:49:51.752070 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:49:51.752034    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" path="/var/lib/kubelet/pods/45866176-c1f4-47d2-a2c0-ceaa41379540/volumes"
Jul 24 00:50:00.491850 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.491817    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"]
Jul 24 00:50:00.492346 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.492075    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np" podUID="aaaac91d-d17b-4aaa-852b-809a780ed638" containerName="maas-api" containerID="cri-o://9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4" gracePeriod=30
Jul 24 00:50:00.731292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.731267    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:50:00.887534 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.887504    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4p89\" (UniqueName: \"kubernetes.io/projected/aaaac91d-d17b-4aaa-852b-809a780ed638-kube-api-access-t4p89\") pod \"aaaac91d-d17b-4aaa-852b-809a780ed638\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") "
Jul 24 00:50:00.887688 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.887602    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaaac91d-d17b-4aaa-852b-809a780ed638-openshift-service-ca\") pod \"aaaac91d-d17b-4aaa-852b-809a780ed638\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") "
Jul 24 00:50:00.887733 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.887706    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaaac91d-d17b-4aaa-852b-809a780ed638-maas-api-tls\") pod \"aaaac91d-d17b-4aaa-852b-809a780ed638\" (UID: \"aaaac91d-d17b-4aaa-852b-809a780ed638\") "
Jul 24 00:50:00.888002 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.887971    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaaac91d-d17b-4aaa-852b-809a780ed638-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aaaac91d-d17b-4aaa-852b-809a780ed638" (UID: "aaaac91d-d17b-4aaa-852b-809a780ed638"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:50:00.889477 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.889450    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaaac91d-d17b-4aaa-852b-809a780ed638-kube-api-access-t4p89" (OuterVolumeSpecName: "kube-api-access-t4p89") pod "aaaac91d-d17b-4aaa-852b-809a780ed638" (UID: "aaaac91d-d17b-4aaa-852b-809a780ed638"). InnerVolumeSpecName "kube-api-access-t4p89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:50:00.889719 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.889702    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaaac91d-d17b-4aaa-852b-809a780ed638-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aaaac91d-d17b-4aaa-852b-809a780ed638" (UID: "aaaac91d-d17b-4aaa-852b-809a780ed638"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:50:00.989097 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.989059    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaaac91d-d17b-4aaa-852b-809a780ed638-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:50:00.989097 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.989093    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4p89\" (UniqueName: \"kubernetes.io/projected/aaaac91d-d17b-4aaa-852b-809a780ed638-kube-api-access-t4p89\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:50:00.989097 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:00.989105    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaaac91d-d17b-4aaa-852b-809a780ed638-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:50:01.175674 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:01.175576    2574 generic.go:356] "Generic (PLEG): container finished" podID="aaaac91d-d17b-4aaa-852b-809a780ed638" containerID="9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4" exitCode=0
Jul 24 00:50:01.175674 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:01.175620    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"
Jul 24 00:50:01.175674 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:01.175661    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np" event={"ID":"aaaac91d-d17b-4aaa-852b-809a780ed638","Type":"ContainerDied","Data":"9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4"}
Jul 24 00:50:01.175894 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:01.175714    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np" event={"ID":"aaaac91d-d17b-4aaa-852b-809a780ed638","Type":"ContainerDied","Data":"e00cc27623384524bc81183310372f6453c0bfe361baa0a7459dd1d8bc76ccb7"}
Jul 24 00:50:01.175894 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:01.175737    2574 scope.go:117] "RemoveContainer" containerID="9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4"
Jul 24 00:50:01.184480 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:01.184466    2574 scope.go:117] "RemoveContainer" containerID="9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4"
Jul 24 00:50:01.184680 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:50:01.184663    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4\": container with ID starting with 9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4 not found: ID does not exist" containerID="9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4"
Jul 24 00:50:01.184729 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:01.184687    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4"} err="failed to get container status \"9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4\": rpc error: code = NotFound desc = could not find container \"9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4\": container with ID starting with 9df5c48f88dc8a392d101e6ae7636940bddabf4806678d8fe6b0c77086368ca4 not found: ID does not exist"
Jul 24 00:50:01.196680 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:01.196659    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"]
Jul 24 00:50:01.200863 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:01.200844    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a4c256-6f6c879f95-722np"]
Jul 24 00:50:01.752423 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:01.752383    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaaac91d-d17b-4aaa-852b-809a780ed638" path="/var/lib/kubelet/pods/aaaac91d-d17b-4aaa-852b-809a780ed638/volumes"
Jul 24 00:50:12.401077 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.401047    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"]
Jul 24 00:50:12.401483 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.401326    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerName="istio-proxy"
Jul 24 00:50:12.401483 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.401341    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerName="istio-proxy"
Jul 24 00:50:12.401483 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.401354    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaaac91d-d17b-4aaa-852b-809a780ed638" containerName="maas-api"
Jul 24 00:50:12.401483 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.401360    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaaac91d-d17b-4aaa-852b-809a780ed638" containerName="maas-api"
Jul 24 00:50:12.401483 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.401368    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1623325-a609-479b-9b97-7000897d7eaa" containerName="maas-api"
Jul 24 00:50:12.401483 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.401374    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1623325-a609-479b-9b97-7000897d7eaa" containerName="maas-api"
Jul 24 00:50:12.401483 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.401424    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1623325-a609-479b-9b97-7000897d7eaa" containerName="maas-api"
Jul 24 00:50:12.401483 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.401432    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerName="istio-proxy"
Jul 24 00:50:12.401483 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.401437    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaaac91d-d17b-4aaa-852b-809a780ed638" containerName="maas-api"
Jul 24 00:50:12.405372 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.405356    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.407676 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.407654    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-44952a-openshift-default-dockercfg-hstmb\""
Jul 24 00:50:12.418656 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.418619    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"]
Jul 24 00:50:12.479748 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.479710    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-token\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.479748 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.479746    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18bbdd4d-b45c-40e1-8842-265c126a6dce-istiod-ca-cert\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.479991 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.479809    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-credential-socket\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.479991 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.479847    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-data\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.479991 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.479866    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vkf\" (UniqueName: \"kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-kube-api-access-54vkf\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.479991 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.479917    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-socket\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.479991 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.479959    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-certs\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.479991 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.479985    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-podinfo\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.480243 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.480049    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-envoy\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.580815 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.580775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-credential-socket\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.580815 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.580819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-data\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581104 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.580842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vkf\" (UniqueName: \"kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-kube-api-access-54vkf\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581104 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.580865    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-socket\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581104 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.580886    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-certs\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581104 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.580902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-podinfo\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581104 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.580947    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-envoy\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581104 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.580980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-token\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581104 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.581006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18bbdd4d-b45c-40e1-8842-265c126a6dce-istiod-ca-cert\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581424 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.581243    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-data\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581424 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.581307    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-credential-socket\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581623 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.581597    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-socket\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581692 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.581639    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-certs\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.581728 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.581704    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18bbdd4d-b45c-40e1-8842-265c126a6dce-istiod-ca-cert\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.583437 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.583417    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-envoy\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.583619 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.583599    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-podinfo\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.588551 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.588528    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vkf\" (UniqueName: \"kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-kube-api-access-54vkf\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.588690 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.588667    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-token\") pod \"e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") " pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.717294 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.717220    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:12.848438 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.848415    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"]
Jul 24 00:50:12.850838 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:50:12.850813    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bbdd4d_b45c_40e1_8842_265c126a6dce.slice/crio-31685c2191bde5daa495282ff84d9cb813f2aad636706a966aba75acda3aa455 WatchSource:0}: Error finding container 31685c2191bde5daa495282ff84d9cb813f2aad636706a966aba75acda3aa455: Status 404 returned error can't find the container with id 31685c2191bde5daa495282ff84d9cb813f2aad636706a966aba75acda3aa455
Jul 24 00:50:12.852879 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.852846    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:50:12.852960 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.852912    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:50:12.852960 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:12.852942    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:50:13.223302 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:13.223271    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" event={"ID":"18bbdd4d-b45c-40e1-8842-265c126a6dce","Type":"ContainerStarted","Data":"b3effc22cb35e488d4924d24f79b81ffd32c052c289781b5a1cc6c3f66598f41"}
Jul 24 00:50:13.223491 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:13.223332    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" event={"ID":"18bbdd4d-b45c-40e1-8842-265c126a6dce","Type":"ContainerStarted","Data":"31685c2191bde5daa495282ff84d9cb813f2aad636706a966aba75acda3aa455"}
Jul 24 00:50:13.246912 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:13.246855    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" podStartSLOduration=1.246839595 podStartE2EDuration="1.246839595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:50:13.245638583 +0000 UTC m=+2376.093957334" watchObservedRunningTime="2026-07-24 00:50:13.246839595 +0000 UTC m=+2376.095158335"
Jul 24 00:50:13.717421 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:13.717377    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:13.722153 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:13.722126    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:14.227843 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:14.227815    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:14.228736 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:14.228717    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:15.447465 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.447426    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"]
Jul 24 00:50:15.447848 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.447742    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerName="istio-proxy"
Jul 24 00:50:15.447848 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.447755    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerName="istio-proxy"
Jul 24 00:50:15.447848 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.447822    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="45866176-c1f4-47d2-a2c0-ceaa41379540" containerName="istio-proxy"
Jul 24 00:50:15.450455 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.450437    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:15.459103 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.459081    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 00:50:15.459217 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.459082    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-44952a-serving-cert\""
Jul 24 00:50:15.459847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.459830    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nv7dj\""
Jul 24 00:50:15.459847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.459842    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 00:50:15.467457 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.467430    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"]
Jul 24 00:50:15.505504 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.505473    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84641ab4-14d6-4523-8aa8-5b865207d6c5-maas-api-tls\") pod \"maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:15.505666 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.505513    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84641ab4-14d6-4523-8aa8-5b865207d6c5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:15.505666 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.505545    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5kvx\" (UniqueName: \"kubernetes.io/projected/84641ab4-14d6-4523-8aa8-5b865207d6c5-kube-api-access-v5kvx\") pod \"maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:15.606710 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.606680    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5kvx\" (UniqueName: \"kubernetes.io/projected/84641ab4-14d6-4523-8aa8-5b865207d6c5-kube-api-access-v5kvx\") pod \"maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:15.606980 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.606744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84641ab4-14d6-4523-8aa8-5b865207d6c5-maas-api-tls\") pod \"maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:15.606980 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.606775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84641ab4-14d6-4523-8aa8-5b865207d6c5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:15.607430 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.607410    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84641ab4-14d6-4523-8aa8-5b865207d6c5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:15.609244 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.609224    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84641ab4-14d6-4523-8aa8-5b865207d6c5-maas-api-tls\") pod \"maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:15.618237 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.618215    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5kvx\" (UniqueName: \"kubernetes.io/projected/84641ab4-14d6-4523-8aa8-5b865207d6c5-kube-api-access-v5kvx\") pod \"maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:15.761829 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.761750    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:15.889248 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:15.889103    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"]
Jul 24 00:50:15.891854 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:50:15.891830    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84641ab4_14d6_4523_8aa8_5b865207d6c5.slice/crio-7b9c166a76f5126d4c4ce888e67ab2ec3c21e8449b21bb26e414917e7a81c447 WatchSource:0}: Error finding container 7b9c166a76f5126d4c4ce888e67ab2ec3c21e8449b21bb26e414917e7a81c447: Status 404 returned error can't find the container with id 7b9c166a76f5126d4c4ce888e67ab2ec3c21e8449b21bb26e414917e7a81c447
Jul 24 00:50:16.236378 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:16.236343    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz" event={"ID":"84641ab4-14d6-4523-8aa8-5b865207d6c5","Type":"ContainerStarted","Data":"7b9c166a76f5126d4c4ce888e67ab2ec3c21e8449b21bb26e414917e7a81c447"}
Jul 24 00:50:17.241417 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:17.241381    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz" event={"ID":"84641ab4-14d6-4523-8aa8-5b865207d6c5","Type":"ContainerStarted","Data":"e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0"}
Jul 24 00:50:17.241863 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:17.241552    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:17.260396 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:17.260346    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz" podStartSLOduration=1.7823002350000001 podStartE2EDuration="2.260332125s" totalImagesPullingTime="478.03189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:50:17.259235167 +0000 UTC m=+2380.107553908" watchObservedRunningTime="2026-07-24 00:50:17.260332125 +0000 UTC m=+2380.108650858"
Jul 24 00:50:23.251756 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:23.251726    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 00:50:24.277476 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.277435    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"]
Jul 24 00:50:24.280806 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.280789    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 00:50:24.284682 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.284661    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-91a75b\""
Jul 24 00:50:24.291935 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.291912    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"]
Jul 24 00:50:24.383875 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.383833    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwm5x\" (UniqueName: \"kubernetes.io/projected/ea465a24-914a-4b7c-af9d-12e7865db575-kube-api-access-mwm5x\") pod \"payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc\" (UID: \"ea465a24-914a-4b7c-af9d-12e7865db575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 00:50:24.384067 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.383966    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea465a24-914a-4b7c-af9d-12e7865db575-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc\" (UID: \"ea465a24-914a-4b7c-af9d-12e7865db575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 00:50:24.484468 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.484419    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwm5x\" (UniqueName: \"kubernetes.io/projected/ea465a24-914a-4b7c-af9d-12e7865db575-kube-api-access-mwm5x\") pod \"payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc\" (UID: \"ea465a24-914a-4b7c-af9d-12e7865db575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 00:50:24.484657 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.484508    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea465a24-914a-4b7c-af9d-12e7865db575-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc\" (UID: \"ea465a24-914a-4b7c-af9d-12e7865db575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 00:50:24.485192 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.485170    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea465a24-914a-4b7c-af9d-12e7865db575-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc\" (UID: \"ea465a24-914a-4b7c-af9d-12e7865db575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 00:50:24.493733 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.493704    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwm5x\" (UniqueName: \"kubernetes.io/projected/ea465a24-914a-4b7c-af9d-12e7865db575-kube-api-access-mwm5x\") pod \"payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc\" (UID: \"ea465a24-914a-4b7c-af9d-12e7865db575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 00:50:24.590846 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.590804    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 00:50:24.741916 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:24.741832    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"]
Jul 24 00:50:24.769362 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:50:24.769327    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea465a24_914a_4b7c_af9d_12e7865db575.slice/crio-e295e1a501e9cfae0c6725c36a53ef53a49f49a35894200757b21a94530ec789 WatchSource:0}: Error finding container e295e1a501e9cfae0c6725c36a53ef53a49f49a35894200757b21a94530ec789: Status 404 returned error can't find the container with id e295e1a501e9cfae0c6725c36a53ef53a49f49a35894200757b21a94530ec789
Jul 24 00:50:25.274938 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:25.274903    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc" event={"ID":"ea465a24-914a-4b7c-af9d-12e7865db575","Type":"ContainerStarted","Data":"b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b"}
Jul 24 00:50:25.275128 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:25.274957    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc" event={"ID":"ea465a24-914a-4b7c-af9d-12e7865db575","Type":"ContainerStarted","Data":"e295e1a501e9cfae0c6725c36a53ef53a49f49a35894200757b21a94530ec789"}
Jul 24 00:50:25.275128 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:25.274988    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 00:50:25.293934 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:25.293887    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc" podStartSLOduration=1.293870788 podStartE2EDuration="1.293870788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:50:25.293366309 +0000 UTC m=+2388.141685052" watchObservedRunningTime="2026-07-24 00:50:25.293870788 +0000 UTC m=+2388.142189528"
Jul 24 00:50:30.241921 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:30.241367    2574 patch_prober.go:28] interesting pod/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 24 00:50:30.241921 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:30.241435    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 00:50:30.295726 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:30.295696    2574 generic.go:356] "Generic (PLEG): container finished" podID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerID="b3effc22cb35e488d4924d24f79b81ffd32c052c289781b5a1cc6c3f66598f41" exitCode=0
Jul 24 00:50:30.295840 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:30.295770    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" event={"ID":"18bbdd4d-b45c-40e1-8842-265c126a6dce","Type":"ContainerDied","Data":"b3effc22cb35e488d4924d24f79b81ffd32c052c289781b5a1cc6c3f66598f41"}
Jul 24 00:50:30.296224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:30.296204    2574 scope.go:117] "RemoveContainer" containerID="b3effc22cb35e488d4924d24f79b81ffd32c052c289781b5a1cc6c3f66598f41"
Jul 24 00:50:30.297187 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:30.297159    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:50:30.297270 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:30.297245    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:50:30.297328 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:30.297277    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:50:31.301830 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:31.301798    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" event={"ID":"18bbdd4d-b45c-40e1-8842-265c126a6dce","Type":"ContainerStarted","Data":"7153c017d59862ad32ffeaeb82add09c16db6ef44844386ad31072f8bf19e72c"}
Jul 24 00:50:31.717558 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:31.717523    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:31.717805 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:31.717783    2574 patch_prober.go:28] interesting pod/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 24 00:50:31.717909 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:31.717850    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 24 00:50:32.718126 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:32.718087    2574 patch_prober.go:28] interesting pod/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 24 00:50:32.718559 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:32.718149    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 24 00:50:33.717834 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:33.717798    2574 patch_prober.go:28] interesting pod/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 24 00:50:33.717984 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:33.717853    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 24 00:50:34.717816 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:34.717783    2574 patch_prober.go:28] interesting pod/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 24 00:50:34.718201 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:34.717839    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 24 00:50:35.721281 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:35.721248    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:35.721650 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:35.721605    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:35.722115 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:35.722098    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 00:50:36.280966 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:36.280933    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 00:50:38.516517 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.516482    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"]
Jul 24 00:50:38.519524 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.519507    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.522223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.522191    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-44952a\"/\"openshift-service-ca.crt\""
Jul 24 00:50:38.522223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.522220    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-44952a\"/\"kube-root-ca.crt\""
Jul 24 00:50:38.523551 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.523522    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-44952a\"/\"auth-test-model-44952a-kserve-self-signed-certs\""
Jul 24 00:50:38.523664 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.523598    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-44952a\"/\"default-dockercfg-hn7tj\""
Jul 24 00:50:38.527402 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.527377    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"]
Jul 24 00:50:38.611666 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.611640    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-model-cache\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.611793 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.611674    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-home\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.611793 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.611718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a605f9-d439-461b-9955-2ad0aeadfe27-tls-certs\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.611793 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.611740    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsp7m\" (UniqueName: \"kubernetes.io/projected/67a605f9-d439-461b-9955-2ad0aeadfe27-kube-api-access-zsp7m\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.611793 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.611787    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-dshm\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.611934 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.611835    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-tmp-dir\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.713079 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.713051    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-home\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.713079 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.713082    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a605f9-d439-461b-9955-2ad0aeadfe27-tls-certs\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.713338 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.713101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsp7m\" (UniqueName: \"kubernetes.io/projected/67a605f9-d439-461b-9955-2ad0aeadfe27-kube-api-access-zsp7m\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.713338 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.713134    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-dshm\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.713338 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.713151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-tmp-dir\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.713338 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.713196    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-model-cache\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.713624 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.713601    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-home\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.713692 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.713624    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-model-cache\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.713692 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.713668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-tmp-dir\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.715608 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.715586    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-dshm\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.715690 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.715648    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a605f9-d439-461b-9955-2ad0aeadfe27-tls-certs\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.720463 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.720442    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsp7m\" (UniqueName: \"kubernetes.io/projected/67a605f9-d439-461b-9955-2ad0aeadfe27-kube-api-access-zsp7m\") pod \"auth-test-model-44952a-kserve-64548fc99-6ggbp\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") " pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.829196 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.829168    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:38.952991 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:38.952966    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"]
Jul 24 00:50:38.955463 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:50:38.955428    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67a605f9_d439_461b_9955_2ad0aeadfe27.slice/crio-b009025d37ea92ae74d12aed98e86f270e4acccdcfcbba54fe77db4db1f818a6 WatchSource:0}: Error finding container b009025d37ea92ae74d12aed98e86f270e4acccdcfcbba54fe77db4db1f818a6: Status 404 returned error can't find the container with id b009025d37ea92ae74d12aed98e86f270e4acccdcfcbba54fe77db4db1f818a6
Jul 24 00:50:39.340256 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:39.340212    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp" event={"ID":"67a605f9-d439-461b-9955-2ad0aeadfe27","Type":"ContainerStarted","Data":"0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f"}
Jul 24 00:50:39.340451 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:39.340264    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp" event={"ID":"67a605f9-d439-461b-9955-2ad0aeadfe27","Type":"ContainerStarted","Data":"b009025d37ea92ae74d12aed98e86f270e4acccdcfcbba54fe77db4db1f818a6"}
Jul 24 00:50:39.356515 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:39.356461    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp" podStartSLOduration=1.35644287 podStartE2EDuration="1.35644287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:50:39.356430394 +0000 UTC m=+2402.204749135" watchObservedRunningTime="2026-07-24 00:50:39.35644287 +0000 UTC m=+2402.204761611"
Jul 24 00:50:48.829453 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:48.829407    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:48.829453 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:48.829464    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:48.841665 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:48.841643    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:50:49.388711 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:50:49.388672    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:51:09.529372 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.529337    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"]
Jul 24 00:51:09.533688 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.533664    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.536176 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.536154    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"openshift-service-ca.crt\""
Jul 24 00:51:09.537289 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.537272    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"kube-root-ca.crt\""
Jul 24 00:51:09.537586 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.537569    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"default-dockercfg-7vpfr\""
Jul 24 00:51:09.537633 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.537573    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"auth-test-model-91a75b-kserve-self-signed-certs\""
Jul 24 00:51:09.543452 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.543432    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"]
Jul 24 00:51:09.587476 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.587452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-home\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.587591 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.587486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-dshm\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.587591 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.587516    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tmp-dir\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.587669 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.587604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9br\" (UniqueName: \"kubernetes.io/projected/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-kube-api-access-sx9br\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.587669 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.587641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-model-cache\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.587669 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.587664    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tls-certs\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.688162 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.688137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-home\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.688162 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.688164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-dshm\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.688393 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.688185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tmp-dir\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.688393 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.688217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9br\" (UniqueName: \"kubernetes.io/projected/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-kube-api-access-sx9br\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.688393 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.688250    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-model-cache\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.688393 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.688374    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tls-certs\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.688592 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.688569    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-home\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.688682 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.688660    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tmp-dir\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.688727 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.688688    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-model-cache\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.690458 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.690438    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-dshm\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.690822 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.690800    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tls-certs\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.698187 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.698157    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9br\" (UniqueName: \"kubernetes.io/projected/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-kube-api-access-sx9br\") pod \"auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") " pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.844390 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.844359    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:09.971658 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:09.971621    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"]
Jul 24 00:51:09.974451 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:51:09.974423    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7e02456_1bfb_4083_8cf3_2ebc3ac104cd.slice/crio-3de2e35421a97bbd99f29857dfc62c169a091d26054bfacc90e05f8758cf5889 WatchSource:0}: Error finding container 3de2e35421a97bbd99f29857dfc62c169a091d26054bfacc90e05f8758cf5889: Status 404 returned error can't find the container with id 3de2e35421a97bbd99f29857dfc62c169a091d26054bfacc90e05f8758cf5889
Jul 24 00:51:10.467347 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:10.467300    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd" event={"ID":"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd","Type":"ContainerStarted","Data":"87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149"}
Jul 24 00:51:10.467347 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:10.467353    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd" event={"ID":"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd","Type":"ContainerStarted","Data":"3de2e35421a97bbd99f29857dfc62c169a091d26054bfacc90e05f8758cf5889"}
Jul 24 00:51:10.487478 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:10.487428    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd" podStartSLOduration=1.487412688 podStartE2EDuration="1.487412688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:51:10.486433158 +0000 UTC m=+2433.334751899" watchObservedRunningTime="2026-07-24 00:51:10.487412688 +0000 UTC m=+2433.335731428"
Jul 24 00:51:19.844664 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:19.844631    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:19.845088 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:19.844678    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:19.856928 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:19.856900    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:51:20.521603 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:51:20.521574    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:53:10.743123 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:10.743082    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"]
Jul 24 00:53:11.634858 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:11.634826    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"]
Jul 24 00:53:25.129008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.128974    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"]
Jul 24 00:53:25.132222 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.132204    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.134744 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.134727    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-44952a\"/\"sub-test-model-44952a-kserve-self-signed-certs\""
Jul 24 00:53:25.142726 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.142704    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"]
Jul 24 00:53:25.260699 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.260663    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-home\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.260868 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.260716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nspgm\" (UniqueName: \"kubernetes.io/projected/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-kube-api-access-nspgm\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.260868 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.260796    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-model-cache\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.260868 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.260846    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tls-certs\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.261012 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.260919    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-dshm\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.261012 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.260951    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tmp-dir\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.362008 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.361972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-home\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.362180 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.362016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nspgm\" (UniqueName: \"kubernetes.io/projected/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-kube-api-access-nspgm\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.362180 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.362145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-model-cache\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.362408 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.362204    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tls-certs\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.362408 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.362258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-dshm\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.362408 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.362295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tmp-dir\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.362573 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.362411    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-home\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.362573 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.362514    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-model-cache\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.362689 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.362671    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tmp-dir\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.364446 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.364416    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-dshm\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.364666 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.364646    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tls-certs\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.370124 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.370102    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nspgm\" (UniqueName: \"kubernetes.io/projected/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-kube-api-access-nspgm\") pod \"sub-test-model-44952a-kserve-79bc5fc88f-dk7kr\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") " pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.442029 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.441952    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:25.570036 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.570005    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"]
Jul 24 00:53:25.572844 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:53:25.572813    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d9ecf74_2f52_4037_ad45_06c13ae9cce6.slice/crio-a433f53b1b4caecb7ef94acdb47a1f37e3cdcf6fbd69c5d2fd6048b7f512ded3 WatchSource:0}: Error finding container a433f53b1b4caecb7ef94acdb47a1f37e3cdcf6fbd69c5d2fd6048b7f512ded3: Status 404 returned error can't find the container with id a433f53b1b4caecb7ef94acdb47a1f37e3cdcf6fbd69c5d2fd6048b7f512ded3
Jul 24 00:53:25.761349 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.761255    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp" podUID="67a605f9-d439-461b-9955-2ad0aeadfe27" containerName="main" containerID="cri-o://0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f" gracePeriod=45
Jul 24 00:53:25.995719 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:25.995695    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:53:26.037350 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.037255    2574 generic.go:356] "Generic (PLEG): container finished" podID="67a605f9-d439-461b-9955-2ad0aeadfe27" containerID="0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f" exitCode=0
Jul 24 00:53:26.037350 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.037297    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"
Jul 24 00:53:26.037577 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.037345    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp" event={"ID":"67a605f9-d439-461b-9955-2ad0aeadfe27","Type":"ContainerDied","Data":"0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f"}
Jul 24 00:53:26.037577 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.037394    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp" event={"ID":"67a605f9-d439-461b-9955-2ad0aeadfe27","Type":"ContainerDied","Data":"b009025d37ea92ae74d12aed98e86f270e4acccdcfcbba54fe77db4db1f818a6"}
Jul 24 00:53:26.037577 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.037421    2574 scope.go:117] "RemoveContainer" containerID="0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f"
Jul 24 00:53:26.046009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.045978    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr" event={"ID":"5d9ecf74-2f52-4037-ad45-06c13ae9cce6","Type":"ContainerStarted","Data":"48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661"}
Jul 24 00:53:26.046176 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.046147    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr" event={"ID":"5d9ecf74-2f52-4037-ad45-06c13ae9cce6","Type":"ContainerStarted","Data":"a433f53b1b4caecb7ef94acdb47a1f37e3cdcf6fbd69c5d2fd6048b7f512ded3"}
Jul 24 00:53:26.047843 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.047825    2574 scope.go:117] "RemoveContainer" containerID="0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f"
Jul 24 00:53:26.048088 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:53:26.048068    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f\": container with ID starting with 0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f not found: ID does not exist" containerID="0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f"
Jul 24 00:53:26.048172 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.048097    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f"} err="failed to get container status \"0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f\": rpc error: code = NotFound desc = could not find container \"0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f\": container with ID starting with 0c086d3d8152dc42402bbc5e05889b66518beb4051ee25360ed8e6c1de683a8f not found: ID does not exist"
Jul 24 00:53:26.063836 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.063797    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr" podStartSLOduration=1.063785392 podStartE2EDuration="1.063785392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:53:26.061402122 +0000 UTC m=+2568.909720877" watchObservedRunningTime="2026-07-24 00:53:26.063785392 +0000 UTC m=+2568.912104253"
Jul 24 00:53:26.170292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170264    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-dshm\") pod \"67a605f9-d439-461b-9955-2ad0aeadfe27\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") "
Jul 24 00:53:26.170762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170338    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a605f9-d439-461b-9955-2ad0aeadfe27-tls-certs\") pod \"67a605f9-d439-461b-9955-2ad0aeadfe27\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") "
Jul 24 00:53:26.170762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170369    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-tmp-dir\") pod \"67a605f9-d439-461b-9955-2ad0aeadfe27\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") "
Jul 24 00:53:26.170762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170424    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-model-cache\") pod \"67a605f9-d439-461b-9955-2ad0aeadfe27\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") "
Jul 24 00:53:26.170762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170525    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-home\") pod \"67a605f9-d439-461b-9955-2ad0aeadfe27\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") "
Jul 24 00:53:26.170762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170572    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsp7m\" (UniqueName: \"kubernetes.io/projected/67a605f9-d439-461b-9955-2ad0aeadfe27-kube-api-access-zsp7m\") pod \"67a605f9-d439-461b-9955-2ad0aeadfe27\" (UID: \"67a605f9-d439-461b-9955-2ad0aeadfe27\") "
Jul 24 00:53:26.170762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170645    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-model-cache" (OuterVolumeSpecName: "model-cache") pod "67a605f9-d439-461b-9955-2ad0aeadfe27" (UID: "67a605f9-d439-461b-9955-2ad0aeadfe27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:53:26.170762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170674    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67a605f9-d439-461b-9955-2ad0aeadfe27" (UID: "67a605f9-d439-461b-9955-2ad0aeadfe27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:53:26.171163 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170781    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-home" (OuterVolumeSpecName: "home") pod "67a605f9-d439-461b-9955-2ad0aeadfe27" (UID: "67a605f9-d439-461b-9955-2ad0aeadfe27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:53:26.171163 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170908    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:26.171163 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170929    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:26.171163 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.170944    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:26.172750 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.172722    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a605f9-d439-461b-9955-2ad0aeadfe27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67a605f9-d439-461b-9955-2ad0aeadfe27" (UID: "67a605f9-d439-461b-9955-2ad0aeadfe27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:53:26.172862 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.172754    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-dshm" (OuterVolumeSpecName: "dshm") pod "67a605f9-d439-461b-9955-2ad0aeadfe27" (UID: "67a605f9-d439-461b-9955-2ad0aeadfe27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:53:26.172912 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.172872    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a605f9-d439-461b-9955-2ad0aeadfe27-kube-api-access-zsp7m" (OuterVolumeSpecName: "kube-api-access-zsp7m") pod "67a605f9-d439-461b-9955-2ad0aeadfe27" (UID: "67a605f9-d439-461b-9955-2ad0aeadfe27"). InnerVolumeSpecName "kube-api-access-zsp7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:53:26.271379 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.271342    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsp7m\" (UniqueName: \"kubernetes.io/projected/67a605f9-d439-461b-9955-2ad0aeadfe27-kube-api-access-zsp7m\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:26.271379 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.271371    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a605f9-d439-461b-9955-2ad0aeadfe27-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:26.271379 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.271382    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a605f9-d439-461b-9955-2ad0aeadfe27-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:26.359090 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.359062    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"]
Jul 24 00:53:26.362767 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.362743    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/auth-test-model-44952a-kserve-64548fc99-6ggbp"]
Jul 24 00:53:26.653878 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.653763    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd" podUID="b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" containerName="main" containerID="cri-o://87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149" gracePeriod=45
Jul 24 00:53:26.896649 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:26.896626    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:53:27.052950 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.052865    2574 generic.go:356] "Generic (PLEG): container finished" podID="b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" containerID="87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149" exitCode=0
Jul 24 00:53:27.052950 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.052906    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"
Jul 24 00:53:27.053154 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.052953    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd" event={"ID":"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd","Type":"ContainerDied","Data":"87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149"}
Jul 24 00:53:27.053154 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.052995    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd" event={"ID":"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd","Type":"ContainerDied","Data":"3de2e35421a97bbd99f29857dfc62c169a091d26054bfacc90e05f8758cf5889"}
Jul 24 00:53:27.053154 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.053018    2574 scope.go:117] "RemoveContainer" containerID="87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149"
Jul 24 00:53:27.060741 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.060722    2574 scope.go:117] "RemoveContainer" containerID="87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149"
Jul 24 00:53:27.061015 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:53:27.060990    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149\": container with ID starting with 87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149 not found: ID does not exist" containerID="87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149"
Jul 24 00:53:27.061113 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.061018    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149"} err="failed to get container status \"87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149\": rpc error: code = NotFound desc = could not find container \"87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149\": container with ID starting with 87dcaa1ee58afc4f8159baac2a4f8468ec1bc336c3ef455058533360ead39149 not found: ID does not exist"
Jul 24 00:53:27.077456 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077434    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-model-cache\") pod \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") "
Jul 24 00:53:27.077537 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077505    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx9br\" (UniqueName: \"kubernetes.io/projected/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-kube-api-access-sx9br\") pod \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") "
Jul 24 00:53:27.077592 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077536    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-home\") pod \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") "
Jul 24 00:53:27.077592 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077565    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tmp-dir\") pod \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") "
Jul 24 00:53:27.077592 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077589    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-dshm\") pod \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") "
Jul 24 00:53:27.077752 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077611    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tls-certs\") pod \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\" (UID: \"b7e02456-1bfb-4083-8cf3-2ebc3ac104cd\") "
Jul 24 00:53:27.077752 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077671    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" (UID: "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:53:27.077862 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077806    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-home" (OuterVolumeSpecName: "home") pod "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" (UID: "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:53:27.077862 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077848    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" (UID: "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:53:27.077959 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077906    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:27.077959 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077924    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:27.077959 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.077938    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:27.079800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.079770    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-dshm" (OuterVolumeSpecName: "dshm") pod "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" (UID: "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:53:27.079888 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.079780    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-kube-api-access-sx9br" (OuterVolumeSpecName: "kube-api-access-sx9br") pod "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" (UID: "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd"). InnerVolumeSpecName "kube-api-access-sx9br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:53:27.079888 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.079813    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" (UID: "b7e02456-1bfb-4083-8cf3-2ebc3ac104cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:53:27.178578 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.178548    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9br\" (UniqueName: \"kubernetes.io/projected/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-kube-api-access-sx9br\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:27.178578 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.178571    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:27.178578 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.178580    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:53:27.373455 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.373423    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"]
Jul 24 00:53:27.375876 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.375851    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/auth-test-model-91a75b-kserve-5cdc4bcdd7-t7pmd"]
Jul 24 00:53:27.752086 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.751988    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a605f9-d439-461b-9955-2ad0aeadfe27" path="/var/lib/kubelet/pods/67a605f9-d439-461b-9955-2ad0aeadfe27/volumes"
Jul 24 00:53:27.752499 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:27.752478    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" path="/var/lib/kubelet/pods/b7e02456-1bfb-4083-8cf3-2ebc3ac104cd/volumes"
Jul 24 00:53:35.443095 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:35.443057    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:35.443095 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:35.443105    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:35.455463 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:35.455434    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:53:36.103046 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:53:36.103019    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:54:04.643621 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.643542    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"]
Jul 24 00:54:04.644223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.643984    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a605f9-d439-461b-9955-2ad0aeadfe27" containerName="main"
Jul 24 00:54:04.644223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.644004    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a605f9-d439-461b-9955-2ad0aeadfe27" containerName="main"
Jul 24 00:54:04.644223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.644024    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" containerName="main"
Jul 24 00:54:04.644223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.644032    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" containerName="main"
Jul 24 00:54:04.644223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.644128    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e02456-1bfb-4083-8cf3-2ebc3ac104cd" containerName="main"
Jul 24 00:54:04.644223 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.644138    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a605f9-d439-461b-9955-2ad0aeadfe27" containerName="main"
Jul 24 00:54:04.647129 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.647110    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.650748 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.650726    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"sub-test-model-91a75b-kserve-self-signed-certs\""
Jul 24 00:54:04.650867 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.650743    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"kube-root-ca.crt\""
Jul 24 00:54:04.650867 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.650726    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"openshift-service-ca.crt\""
Jul 24 00:54:04.650867 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.650777    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"default-dockercfg-7vpfr\""
Jul 24 00:54:04.665302 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.665279    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"]
Jul 24 00:54:04.803836 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.803794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7g7x\" (UniqueName: \"kubernetes.io/projected/d2683f41-3bdd-4690-a11b-558b8a07deb9-kube-api-access-n7g7x\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.803997 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.803884    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-tmp-dir\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.803997 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.803921    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-home\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.804085 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.803997    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-model-cache\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.804085 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.804031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-dshm\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.804085 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.804059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2683f41-3bdd-4690-a11b-558b8a07deb9-tls-certs\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.905333 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.905220    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7g7x\" (UniqueName: \"kubernetes.io/projected/d2683f41-3bdd-4690-a11b-558b8a07deb9-kube-api-access-n7g7x\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.905333 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.905278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-tmp-dir\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.905333 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.905332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-home\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.905620 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.905369    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-model-cache\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.905620 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.905395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-dshm\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.905620 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.905419    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2683f41-3bdd-4690-a11b-558b8a07deb9-tls-certs\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.905782 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.905667    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-home\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.905782 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.905721    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-tmp-dir\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.905876 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.905788    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-model-cache\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.907565 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.907547    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-dshm\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.907897 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.907878    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2683f41-3bdd-4690-a11b-558b8a07deb9-tls-certs\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.913014 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.912992    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7g7x\" (UniqueName: \"kubernetes.io/projected/d2683f41-3bdd-4690-a11b-558b8a07deb9-kube-api-access-n7g7x\") pod \"sub-test-model-91a75b-kserve-84b94c6ff7-666pk\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") " pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:04.956981 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:04.956956    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:05.099748 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:05.099719    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"]
Jul 24 00:54:05.102405 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:54:05.102377    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2683f41_3bdd_4690_a11b_558b8a07deb9.slice/crio-6cb1699c2a76578545c3337fda7bab1a904c7806d434e35d613b2b5f163b608c WatchSource:0}: Error finding container 6cb1699c2a76578545c3337fda7bab1a904c7806d434e35d613b2b5f163b608c: Status 404 returned error can't find the container with id 6cb1699c2a76578545c3337fda7bab1a904c7806d434e35d613b2b5f163b608c
Jul 24 00:54:05.214481 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:05.214397    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk" event={"ID":"d2683f41-3bdd-4690-a11b-558b8a07deb9","Type":"ContainerStarted","Data":"0f68d3848d9c3bf90ef4fc07ac5b0b013983e6759f93f9baaf5e0b14baec4d29"}
Jul 24 00:54:05.214481 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:05.214445    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk" event={"ID":"d2683f41-3bdd-4690-a11b-558b8a07deb9","Type":"ContainerStarted","Data":"6cb1699c2a76578545c3337fda7bab1a904c7806d434e35d613b2b5f163b608c"}
Jul 24 00:54:05.243089 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:05.243037    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk" podStartSLOduration=1.243022129 podStartE2EDuration="1.243022129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:54:05.241621526 +0000 UTC m=+2608.089940268" watchObservedRunningTime="2026-07-24 00:54:05.243022129 +0000 UTC m=+2608.091340870"
Jul 24 00:54:14.957770 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:14.957722    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:14.958389 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:14.957789    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:14.970206 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:14.970182    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:15.267258 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:15.267173    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:54:49.337439 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:49.337401    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"]
Jul 24 00:54:50.317409 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:54:50.317377    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"]
Jul 24 00:55:04.356244 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.356192    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr" podUID="5d9ecf74-2f52-4037-ad45-06c13ae9cce6" containerName="main" containerID="cri-o://48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661" gracePeriod=45
Jul 24 00:55:04.601860 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.601834    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:55:04.726525 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.726439    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nspgm\" (UniqueName: \"kubernetes.io/projected/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-kube-api-access-nspgm\") pod \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") "
Jul 24 00:55:04.726525 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.726477    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-home\") pod \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") "
Jul 24 00:55:04.726525 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.726507    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tls-certs\") pod \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") "
Jul 24 00:55:04.726793 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.726532    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-model-cache\") pod \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") "
Jul 24 00:55:04.726793 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.726554    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tmp-dir\") pod \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") "
Jul 24 00:55:04.726793 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.726607    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-dshm\") pod \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\" (UID: \"5d9ecf74-2f52-4037-ad45-06c13ae9cce6\") "
Jul 24 00:55:04.726793 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.726727    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-home" (OuterVolumeSpecName: "home") pod "5d9ecf74-2f52-4037-ad45-06c13ae9cce6" (UID: "5d9ecf74-2f52-4037-ad45-06c13ae9cce6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:04.726966 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.726828    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d9ecf74-2f52-4037-ad45-06c13ae9cce6" (UID: "5d9ecf74-2f52-4037-ad45-06c13ae9cce6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:04.726966 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.726847    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d9ecf74-2f52-4037-ad45-06c13ae9cce6" (UID: "5d9ecf74-2f52-4037-ad45-06c13ae9cce6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:04.726966 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.726873    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:04.728673 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.728651    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-dshm" (OuterVolumeSpecName: "dshm") pod "5d9ecf74-2f52-4037-ad45-06c13ae9cce6" (UID: "5d9ecf74-2f52-4037-ad45-06c13ae9cce6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:04.728782 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.728719    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d9ecf74-2f52-4037-ad45-06c13ae9cce6" (UID: "5d9ecf74-2f52-4037-ad45-06c13ae9cce6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:55:04.728782 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.728749    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-kube-api-access-nspgm" (OuterVolumeSpecName: "kube-api-access-nspgm") pod "5d9ecf74-2f52-4037-ad45-06c13ae9cce6" (UID: "5d9ecf74-2f52-4037-ad45-06c13ae9cce6"). InnerVolumeSpecName "kube-api-access-nspgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:55:04.827724 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.827699    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:04.827724 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.827721    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nspgm\" (UniqueName: \"kubernetes.io/projected/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-kube-api-access-nspgm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:04.827876 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.827732    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:04.827876 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.827744    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:04.827876 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:04.827769    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9ecf74-2f52-4037-ad45-06c13ae9cce6-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:05.336164 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.336109    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk" podUID="d2683f41-3bdd-4690-a11b-558b8a07deb9" containerName="main" containerID="cri-o://0f68d3848d9c3bf90ef4fc07ac5b0b013983e6759f93f9baaf5e0b14baec4d29" gracePeriod=45
Jul 24 00:55:05.471755 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.471719    2574 generic.go:356] "Generic (PLEG): container finished" podID="d2683f41-3bdd-4690-a11b-558b8a07deb9" containerID="0f68d3848d9c3bf90ef4fc07ac5b0b013983e6759f93f9baaf5e0b14baec4d29" exitCode=0
Jul 24 00:55:05.472158 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.471746    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk" event={"ID":"d2683f41-3bdd-4690-a11b-558b8a07deb9","Type":"ContainerDied","Data":"0f68d3848d9c3bf90ef4fc07ac5b0b013983e6759f93f9baaf5e0b14baec4d29"}
Jul 24 00:55:05.473116 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.473092    2574 generic.go:356] "Generic (PLEG): container finished" podID="5d9ecf74-2f52-4037-ad45-06c13ae9cce6" containerID="48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661" exitCode=0
Jul 24 00:55:05.473240 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.473137    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr" event={"ID":"5d9ecf74-2f52-4037-ad45-06c13ae9cce6","Type":"ContainerDied","Data":"48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661"}
Jul 24 00:55:05.473240 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.473146    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"
Jul 24 00:55:05.473240 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.473162    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr" event={"ID":"5d9ecf74-2f52-4037-ad45-06c13ae9cce6","Type":"ContainerDied","Data":"a433f53b1b4caecb7ef94acdb47a1f37e3cdcf6fbd69c5d2fd6048b7f512ded3"}
Jul 24 00:55:05.473240 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.473183    2574 scope.go:117] "RemoveContainer" containerID="48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661"
Jul 24 00:55:05.482494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.482471    2574 scope.go:117] "RemoveContainer" containerID="48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661"
Jul 24 00:55:05.482752 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:55:05.482732    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661\": container with ID starting with 48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661 not found: ID does not exist" containerID="48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661"
Jul 24 00:55:05.482812 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.482760    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661"} err="failed to get container status \"48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661\": rpc error: code = NotFound desc = could not find container \"48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661\": container with ID starting with 48a72ae4a19b7d087368a092be40affb2ff94423d406d46e8c0445ff2fd3b661 not found: ID does not exist"
Jul 24 00:55:05.495240 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.495213    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"]
Jul 24 00:55:05.499966 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.499942    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/sub-test-model-44952a-kserve-79bc5fc88f-dk7kr"]
Jul 24 00:55:05.581058 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.581034    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:55:05.735552 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.735463    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-home\") pod \"d2683f41-3bdd-4690-a11b-558b8a07deb9\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") "
Jul 24 00:55:05.735552 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.735497    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-dshm\") pod \"d2683f41-3bdd-4690-a11b-558b8a07deb9\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") "
Jul 24 00:55:05.735552 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.735524    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2683f41-3bdd-4690-a11b-558b8a07deb9-tls-certs\") pod \"d2683f41-3bdd-4690-a11b-558b8a07deb9\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") "
Jul 24 00:55:05.735821 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.735611    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7g7x\" (UniqueName: \"kubernetes.io/projected/d2683f41-3bdd-4690-a11b-558b8a07deb9-kube-api-access-n7g7x\") pod \"d2683f41-3bdd-4690-a11b-558b8a07deb9\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") "
Jul 24 00:55:05.735821 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.735641    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-tmp-dir\") pod \"d2683f41-3bdd-4690-a11b-558b8a07deb9\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") "
Jul 24 00:55:05.735821 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.735695    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-model-cache\") pod \"d2683f41-3bdd-4690-a11b-558b8a07deb9\" (UID: \"d2683f41-3bdd-4690-a11b-558b8a07deb9\") "
Jul 24 00:55:05.735821 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.735764    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-home" (OuterVolumeSpecName: "home") pod "d2683f41-3bdd-4690-a11b-558b8a07deb9" (UID: "d2683f41-3bdd-4690-a11b-558b8a07deb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:05.736023 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.735928    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2683f41-3bdd-4690-a11b-558b8a07deb9" (UID: "d2683f41-3bdd-4690-a11b-558b8a07deb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:05.736023 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.735960    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2683f41-3bdd-4690-a11b-558b8a07deb9" (UID: "d2683f41-3bdd-4690-a11b-558b8a07deb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:05.736023 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.735999    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:05.736023 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.736018    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:05.737715 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.737692    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2683f41-3bdd-4690-a11b-558b8a07deb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2683f41-3bdd-4690-a11b-558b8a07deb9" (UID: "d2683f41-3bdd-4690-a11b-558b8a07deb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:55:05.737823 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.737716    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-dshm" (OuterVolumeSpecName: "dshm") pod "d2683f41-3bdd-4690-a11b-558b8a07deb9" (UID: "d2683f41-3bdd-4690-a11b-558b8a07deb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:05.737823 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.737770    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2683f41-3bdd-4690-a11b-558b8a07deb9-kube-api-access-n7g7x" (OuterVolumeSpecName: "kube-api-access-n7g7x") pod "d2683f41-3bdd-4690-a11b-558b8a07deb9" (UID: "d2683f41-3bdd-4690-a11b-558b8a07deb9"). InnerVolumeSpecName "kube-api-access-n7g7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:55:05.751943 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.751919    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d9ecf74-2f52-4037-ad45-06c13ae9cce6" path="/var/lib/kubelet/pods/5d9ecf74-2f52-4037-ad45-06c13ae9cce6/volumes"
Jul 24 00:55:05.836945 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.836918    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:05.836945 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.836941    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2683f41-3bdd-4690-a11b-558b8a07deb9-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:05.836945 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.836949    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2683f41-3bdd-4690-a11b-558b8a07deb9-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:05.837115 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:05.836957    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7g7x\" (UniqueName: \"kubernetes.io/projected/d2683f41-3bdd-4690-a11b-558b8a07deb9-kube-api-access-n7g7x\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:55:06.478192 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:06.478164    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"
Jul 24 00:55:06.478697 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:06.478200    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk" event={"ID":"d2683f41-3bdd-4690-a11b-558b8a07deb9","Type":"ContainerDied","Data":"6cb1699c2a76578545c3337fda7bab1a904c7806d434e35d613b2b5f163b608c"}
Jul 24 00:55:06.478697 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:06.478255    2574 scope.go:117] "RemoveContainer" containerID="0f68d3848d9c3bf90ef4fc07ac5b0b013983e6759f93f9baaf5e0b14baec4d29"
Jul 24 00:55:06.494653 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:06.494631    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"]
Jul 24 00:55:06.497949 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:06.497927    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/sub-test-model-91a75b-kserve-84b94c6ff7-666pk"]
Jul 24 00:55:07.756351 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:07.752733    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2683f41-3bdd-4690-a11b-558b8a07deb9" path="/var/lib/kubelet/pods/d2683f41-3bdd-4690-a11b-558b8a07deb9/volumes"
Jul 24 00:55:12.034501 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.034464    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"]
Jul 24 00:55:12.034964 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.034760    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2683f41-3bdd-4690-a11b-558b8a07deb9" containerName="main"
Jul 24 00:55:12.034964 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.034770    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2683f41-3bdd-4690-a11b-558b8a07deb9" containerName="main"
Jul 24 00:55:12.034964 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.034782    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d9ecf74-2f52-4037-ad45-06c13ae9cce6" containerName="main"
Jul 24 00:55:12.034964 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.034788    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d9ecf74-2f52-4037-ad45-06c13ae9cce6" containerName="main"
Jul 24 00:55:12.034964 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.034845    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2683f41-3bdd-4690-a11b-558b8a07deb9" containerName="main"
Jul 24 00:55:12.034964 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.034854    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d9ecf74-2f52-4037-ad45-06c13ae9cce6" containerName="main"
Jul 24 00:55:12.038902 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.038878    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.041751 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.041728    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-44952a\"/\"kube-root-ca.crt\""
Jul 24 00:55:12.042857 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.042837    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-44952a\"/\"openshift-service-ca.crt\""
Jul 24 00:55:12.043006 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.042977    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-44952a\"/\"rate-test-model-44952a-kserve-self-signed-certs\""
Jul 24 00:55:12.043090 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.042986    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-44952a\"/\"default-dockercfg-hn7tj\""
Jul 24 00:55:12.047922 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.047897    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"]
Jul 24 00:55:12.087362 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.087304    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-model-cache\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.087545 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.087393    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-dshm\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.087545 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.087418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vzmx\" (UniqueName: \"kubernetes.io/projected/814c5997-3419-404d-9587-9a74ec718b7c-kube-api-access-7vzmx\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.087545 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.087456    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-tmp-dir\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.087545 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.087482    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814c5997-3419-404d-9587-9a74ec718b7c-tls-certs\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.087545 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.087541    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-home\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.188253 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.188217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-dshm\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.188253 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.188256    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vzmx\" (UniqueName: \"kubernetes.io/projected/814c5997-3419-404d-9587-9a74ec718b7c-kube-api-access-7vzmx\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.188489 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.188290    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-tmp-dir\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.188489 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.188333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814c5997-3419-404d-9587-9a74ec718b7c-tls-certs\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.188489 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.188468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-home\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.188663 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.188576    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-model-cache\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.188719 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.188705    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-tmp-dir\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.188799 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.188778    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-home\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.188880 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.188859    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-model-cache\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.190536 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.190518    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-dshm\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.190769 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.190752    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814c5997-3419-404d-9587-9a74ec718b7c-tls-certs\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.198699 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.198674    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vzmx\" (UniqueName: \"kubernetes.io/projected/814c5997-3419-404d-9587-9a74ec718b7c-kube-api-access-7vzmx\") pod \"rate-test-model-44952a-kserve-559958f87-86l4r\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") " pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.349392 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.349356    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:12.474029 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.473999    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"]
Jul 24 00:55:12.477385 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:55:12.477359    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod814c5997_3419_404d_9587_9a74ec718b7c.slice/crio-6490f803a6b64f69b476d8f2f4cd24a161fc96112ef8ec8b89b502c36553a298 WatchSource:0}: Error finding container 6490f803a6b64f69b476d8f2f4cd24a161fc96112ef8ec8b89b502c36553a298: Status 404 returned error can't find the container with id 6490f803a6b64f69b476d8f2f4cd24a161fc96112ef8ec8b89b502c36553a298
Jul 24 00:55:12.503798 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:12.503774    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r" event={"ID":"814c5997-3419-404d-9587-9a74ec718b7c","Type":"ContainerStarted","Data":"6490f803a6b64f69b476d8f2f4cd24a161fc96112ef8ec8b89b502c36553a298"}
Jul 24 00:55:13.509811 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:13.509776    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r" event={"ID":"814c5997-3419-404d-9587-9a74ec718b7c","Type":"ContainerStarted","Data":"e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf"}
Jul 24 00:55:13.532823 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:13.532776    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r" podStartSLOduration=1.5327612940000002 podStartE2EDuration="1.532761294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:55:13.530965441 +0000 UTC m=+2676.379284182" watchObservedRunningTime="2026-07-24 00:55:13.532761294 +0000 UTC m=+2676.381080034"
Jul 24 00:55:22.349963 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:22.349925    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:22.349963 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:22.349971    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:22.362439 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:22.362416    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:22.560718 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:22.560683    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:55:50.234089 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.234057    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"]
Jul 24 00:55:50.237212 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.237195    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.239931 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.239906    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"openshift-service-ca.crt\""
Jul 24 00:55:50.240092 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.239907    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"kube-root-ca.crt\""
Jul 24 00:55:50.241154 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.241132    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"rate-test-model-91a75b-kserve-self-signed-certs\""
Jul 24 00:55:50.241154 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.241162    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-91a75b\"/\"default-dockercfg-7vpfr\""
Jul 24 00:55:50.246182 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.246160    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"]
Jul 24 00:55:50.280714 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.280689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-tmp-dir\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.280827 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.280729    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-home\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.280827 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.280752    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8229361c-63c6-4a1f-b575-3989fba8fc7e-tls-certs\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.280905 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.280830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-model-cache\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.280905 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.280866    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-dshm\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.280905 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.280887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w4lw\" (UniqueName: \"kubernetes.io/projected/8229361c-63c6-4a1f-b575-3989fba8fc7e-kube-api-access-5w4lw\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.382040 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.382014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-tmp-dir\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.382144 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.382043    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-home\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.382144 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.382059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8229361c-63c6-4a1f-b575-3989fba8fc7e-tls-certs\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.382144 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.382112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-model-cache\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.382299 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.382158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-dshm\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.382299 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.382179    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w4lw\" (UniqueName: \"kubernetes.io/projected/8229361c-63c6-4a1f-b575-3989fba8fc7e-kube-api-access-5w4lw\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.382431 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.382418    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-home\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.382589 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.382474    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-tmp-dir\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.382662 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.382631    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-model-cache\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.384829 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.384808    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-dshm\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.385204 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.385181    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8229361c-63c6-4a1f-b575-3989fba8fc7e-tls-certs\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.390463 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.390438    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w4lw\" (UniqueName: \"kubernetes.io/projected/8229361c-63c6-4a1f-b575-3989fba8fc7e-kube-api-access-5w4lw\") pod \"rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") " pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.547950 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.547867    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:55:50.674753 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:50.674722    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"]
Jul 24 00:55:50.677887 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:55:50.677860    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8229361c_63c6_4a1f_b575_3989fba8fc7e.slice/crio-4a839804959af8fce08c05a1cb15a48564770bd2e45dea008b9cc43801c3c04a WatchSource:0}: Error finding container 4a839804959af8fce08c05a1cb15a48564770bd2e45dea008b9cc43801c3c04a: Status 404 returned error can't find the container with id 4a839804959af8fce08c05a1cb15a48564770bd2e45dea008b9cc43801c3c04a
Jul 24 00:55:51.670616 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:51.670581    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq" event={"ID":"8229361c-63c6-4a1f-b575-3989fba8fc7e","Type":"ContainerStarted","Data":"fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e"}
Jul 24 00:55:51.670616 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:51.670621    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq" event={"ID":"8229361c-63c6-4a1f-b575-3989fba8fc7e","Type":"ContainerStarted","Data":"4a839804959af8fce08c05a1cb15a48564770bd2e45dea008b9cc43801c3c04a"}
Jul 24 00:55:51.687734 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:55:51.687684    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq" podStartSLOduration=1.687669792 podStartE2EDuration="1.687669792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:55:51.686545663 +0000 UTC m=+2714.534864446" watchObservedRunningTime="2026-07-24 00:55:51.687669792 +0000 UTC m=+2714.535988533"
Jul 24 00:56:00.548782 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:00.548740    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:56:00.549192 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:00.548798    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:56:00.561309 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:00.561279    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:56:00.725010 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:00.724979    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:56:40.502807 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:40.502772    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"]
Jul 24 00:56:41.349021 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:41.348986    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"]
Jul 24 00:56:55.520823 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.520775    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r" podUID="814c5997-3419-404d-9587-9a74ec718b7c" containerName="main" containerID="cri-o://e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf" gracePeriod=45
Jul 24 00:56:55.768665 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.768637    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:56:55.844064 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.844033    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814c5997-3419-404d-9587-9a74ec718b7c-tls-certs\") pod \"814c5997-3419-404d-9587-9a74ec718b7c\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") "
Jul 24 00:56:55.844224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.844071    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-model-cache\") pod \"814c5997-3419-404d-9587-9a74ec718b7c\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") "
Jul 24 00:56:55.844224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.844112    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-tmp-dir\") pod \"814c5997-3419-404d-9587-9a74ec718b7c\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") "
Jul 24 00:56:55.844224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.844141    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-home\") pod \"814c5997-3419-404d-9587-9a74ec718b7c\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") "
Jul 24 00:56:55.844224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.844171    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vzmx\" (UniqueName: \"kubernetes.io/projected/814c5997-3419-404d-9587-9a74ec718b7c-kube-api-access-7vzmx\") pod \"814c5997-3419-404d-9587-9a74ec718b7c\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") "
Jul 24 00:56:55.844224 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.844193    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-dshm\") pod \"814c5997-3419-404d-9587-9a74ec718b7c\" (UID: \"814c5997-3419-404d-9587-9a74ec718b7c\") "
Jul 24 00:56:55.844514 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.844464    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-home" (OuterVolumeSpecName: "home") pod "814c5997-3419-404d-9587-9a74ec718b7c" (UID: "814c5997-3419-404d-9587-9a74ec718b7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:56:55.844579 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.844508    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "814c5997-3419-404d-9587-9a74ec718b7c" (UID: "814c5997-3419-404d-9587-9a74ec718b7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:56:55.844579 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.844496    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "814c5997-3419-404d-9587-9a74ec718b7c" (UID: "814c5997-3419-404d-9587-9a74ec718b7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:56:55.846222 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.846192    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-dshm" (OuterVolumeSpecName: "dshm") pod "814c5997-3419-404d-9587-9a74ec718b7c" (UID: "814c5997-3419-404d-9587-9a74ec718b7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:56:55.846348 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.846242    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/814c5997-3419-404d-9587-9a74ec718b7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "814c5997-3419-404d-9587-9a74ec718b7c" (UID: "814c5997-3419-404d-9587-9a74ec718b7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:56:55.846348 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.846293    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814c5997-3419-404d-9587-9a74ec718b7c-kube-api-access-7vzmx" (OuterVolumeSpecName: "kube-api-access-7vzmx") pod "814c5997-3419-404d-9587-9a74ec718b7c" (UID: "814c5997-3419-404d-9587-9a74ec718b7c"). InnerVolumeSpecName "kube-api-access-7vzmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:56:55.940627 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.940586    2574 generic.go:356] "Generic (PLEG): container finished" podID="814c5997-3419-404d-9587-9a74ec718b7c" containerID="e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf" exitCode=0
Jul 24 00:56:55.940791 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.940650    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"
Jul 24 00:56:55.940791 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.940647    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r" event={"ID":"814c5997-3419-404d-9587-9a74ec718b7c","Type":"ContainerDied","Data":"e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf"}
Jul 24 00:56:55.940791 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.940756    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r" event={"ID":"814c5997-3419-404d-9587-9a74ec718b7c","Type":"ContainerDied","Data":"6490f803a6b64f69b476d8f2f4cd24a161fc96112ef8ec8b89b502c36553a298"}
Jul 24 00:56:55.940791 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.940778    2574 scope.go:117] "RemoveContainer" containerID="e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf"
Jul 24 00:56:55.945722 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.945698    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:55.945722 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.945723    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vzmx\" (UniqueName: \"kubernetes.io/projected/814c5997-3419-404d-9587-9a74ec718b7c-kube-api-access-7vzmx\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:55.945865 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.945734    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:55.945865 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.945742    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814c5997-3419-404d-9587-9a74ec718b7c-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:55.945865 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.945751    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:55.945865 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.945759    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814c5997-3419-404d-9587-9a74ec718b7c-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:55.949454 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.949440    2574 scope.go:117] "RemoveContainer" containerID="e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf"
Jul 24 00:56:55.949701 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:56:55.949683    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf\": container with ID starting with e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf not found: ID does not exist" containerID="e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf"
Jul 24 00:56:55.949752 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.949709    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf"} err="failed to get container status \"e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf\": rpc error: code = NotFound desc = could not find container \"e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf\": container with ID starting with e4b0c36140ba2b68f7735e17023794f790bfe28bdd31ffef3227657785d147bf not found: ID does not exist"
Jul 24 00:56:55.960979 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.960952    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"]
Jul 24 00:56:55.963982 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:55.963960    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-44952a/rate-test-model-44952a-kserve-559958f87-86l4r"]
Jul 24 00:56:56.367521 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.367453    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq" podUID="8229361c-63c6-4a1f-b575-3989fba8fc7e" containerName="main" containerID="cri-o://fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e" gracePeriod=45
Jul 24 00:56:56.623508 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.623437    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:56:56.752641 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.752611    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w4lw\" (UniqueName: \"kubernetes.io/projected/8229361c-63c6-4a1f-b575-3989fba8fc7e-kube-api-access-5w4lw\") pod \"8229361c-63c6-4a1f-b575-3989fba8fc7e\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") "
Jul 24 00:56:56.752814 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.752655    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-tmp-dir\") pod \"8229361c-63c6-4a1f-b575-3989fba8fc7e\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") "
Jul 24 00:56:56.752814 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.752677    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-model-cache\") pod \"8229361c-63c6-4a1f-b575-3989fba8fc7e\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") "
Jul 24 00:56:56.752814 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.752750    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-dshm\") pod \"8229361c-63c6-4a1f-b575-3989fba8fc7e\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") "
Jul 24 00:56:56.752814 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.752808    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8229361c-63c6-4a1f-b575-3989fba8fc7e-tls-certs\") pod \"8229361c-63c6-4a1f-b575-3989fba8fc7e\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") "
Jul 24 00:56:56.753025 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.752837    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-home\") pod \"8229361c-63c6-4a1f-b575-3989fba8fc7e\" (UID: \"8229361c-63c6-4a1f-b575-3989fba8fc7e\") "
Jul 24 00:56:56.753025 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.752914    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8229361c-63c6-4a1f-b575-3989fba8fc7e" (UID: "8229361c-63c6-4a1f-b575-3989fba8fc7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:56:56.753025 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.752992    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "8229361c-63c6-4a1f-b575-3989fba8fc7e" (UID: "8229361c-63c6-4a1f-b575-3989fba8fc7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:56:56.753184 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.753071    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:56.753184 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.753084    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:56.753184 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.753168    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-home" (OuterVolumeSpecName: "home") pod "8229361c-63c6-4a1f-b575-3989fba8fc7e" (UID: "8229361c-63c6-4a1f-b575-3989fba8fc7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:56:56.754886 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.754856    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8229361c-63c6-4a1f-b575-3989fba8fc7e-kube-api-access-5w4lw" (OuterVolumeSpecName: "kube-api-access-5w4lw") pod "8229361c-63c6-4a1f-b575-3989fba8fc7e" (UID: "8229361c-63c6-4a1f-b575-3989fba8fc7e"). InnerVolumeSpecName "kube-api-access-5w4lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:56:56.754886 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.754861    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8229361c-63c6-4a1f-b575-3989fba8fc7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8229361c-63c6-4a1f-b575-3989fba8fc7e" (UID: "8229361c-63c6-4a1f-b575-3989fba8fc7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:56:56.755056 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.754912    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-dshm" (OuterVolumeSpecName: "dshm") pod "8229361c-63c6-4a1f-b575-3989fba8fc7e" (UID: "8229361c-63c6-4a1f-b575-3989fba8fc7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:56:56.854553 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.854512    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8229361c-63c6-4a1f-b575-3989fba8fc7e-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:56.854553 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.854545    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:56.854553 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.854556    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w4lw\" (UniqueName: \"kubernetes.io/projected/8229361c-63c6-4a1f-b575-3989fba8fc7e-kube-api-access-5w4lw\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:56.854762 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.854566    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8229361c-63c6-4a1f-b575-3989fba8fc7e-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:56:56.946838 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.946747    2574 generic.go:356] "Generic (PLEG): container finished" podID="8229361c-63c6-4a1f-b575-3989fba8fc7e" containerID="fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e" exitCode=0
Jul 24 00:56:56.946838 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.946788    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq" event={"ID":"8229361c-63c6-4a1f-b575-3989fba8fc7e","Type":"ContainerDied","Data":"fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e"}
Jul 24 00:56:56.946838 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.946798    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"
Jul 24 00:56:56.946838 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.946812    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq" event={"ID":"8229361c-63c6-4a1f-b575-3989fba8fc7e","Type":"ContainerDied","Data":"4a839804959af8fce08c05a1cb15a48564770bd2e45dea008b9cc43801c3c04a"}
Jul 24 00:56:56.946838 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.946832    2574 scope.go:117] "RemoveContainer" containerID="fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e"
Jul 24 00:56:56.955625 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.955605    2574 scope.go:117] "RemoveContainer" containerID="fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e"
Jul 24 00:56:56.955855 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:56:56.955830    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e\": container with ID starting with fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e not found: ID does not exist" containerID="fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e"
Jul 24 00:56:56.955908 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.955864    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e"} err="failed to get container status \"fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e\": rpc error: code = NotFound desc = could not find container \"fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e\": container with ID starting with fb089cec45bc2040a4b6d8b01753b97e8df73892c77090816abd81d9443e929e not found: ID does not exist"
Jul 24 00:56:56.967910 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.967884    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"]
Jul 24 00:56:56.971292 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:56.971270    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-91a75b/rate-test-model-91a75b-kserve-5ccdf49f8c-pg5dq"]
Jul 24 00:56:57.752274 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:57.752239    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="814c5997-3419-404d-9587-9a74ec718b7c" path="/var/lib/kubelet/pods/814c5997-3419-404d-9587-9a74ec718b7c/volumes"
Jul 24 00:56:57.752663 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:56:57.752649    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8229361c-63c6-4a1f-b575-3989fba8fc7e" path="/var/lib/kubelet/pods/8229361c-63c6-4a1f-b575-3989fba8fc7e/volumes"
Jul 24 00:57:04.555460 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.555378    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"]
Jul 24 00:57:04.555830 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.555773    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="814c5997-3419-404d-9587-9a74ec718b7c" containerName="main"
Jul 24 00:57:04.555830 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.555786    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="814c5997-3419-404d-9587-9a74ec718b7c" containerName="main"
Jul 24 00:57:04.555830 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.555797    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8229361c-63c6-4a1f-b575-3989fba8fc7e" containerName="main"
Jul 24 00:57:04.555830 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.555802    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8229361c-63c6-4a1f-b575-3989fba8fc7e" containerName="main"
Jul 24 00:57:04.555993 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.555859    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8229361c-63c6-4a1f-b575-3989fba8fc7e" containerName="main"
Jul 24 00:57:04.555993 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.555871    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="814c5997-3419-404d-9587-9a74ec718b7c" containerName="main"
Jul 24 00:57:04.560237 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.560216    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:04.562693 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.562666    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-193ff5-serving-cert\""
Jul 24 00:57:04.570696 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.570672    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"]
Jul 24 00:57:04.719914 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.719880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1793562-f014-47a1-9161-c3ae18439fde-maas-api-tls\") pod \"maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:04.720091 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.719925    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqrwz\" (UniqueName: \"kubernetes.io/projected/e1793562-f014-47a1-9161-c3ae18439fde-kube-api-access-vqrwz\") pod \"maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:04.720091 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.720071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1793562-f014-47a1-9161-c3ae18439fde-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:04.821287 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.821257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqrwz\" (UniqueName: \"kubernetes.io/projected/e1793562-f014-47a1-9161-c3ae18439fde-kube-api-access-vqrwz\") pod \"maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:04.821467 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.821370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1793562-f014-47a1-9161-c3ae18439fde-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:04.821467 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.821395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1793562-f014-47a1-9161-c3ae18439fde-maas-api-tls\") pod \"maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:04.822003 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.821973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1793562-f014-47a1-9161-c3ae18439fde-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:04.823800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.823779    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1793562-f014-47a1-9161-c3ae18439fde-maas-api-tls\") pod \"maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:04.829168 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.829145    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqrwz\" (UniqueName: \"kubernetes.io/projected/e1793562-f014-47a1-9161-c3ae18439fde-kube-api-access-vqrwz\") pod \"maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:04.871163 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:04.871127    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:05.003164 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.003138    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"]
Jul 24 00:57:05.005284 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:57:05.005245    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1793562_f014_47a1_9161_c3ae18439fde.slice/crio-edf751045eb8a4bed2bc125385f9cafa79ea0296b21fa97e315f6ba58070bb44 WatchSource:0}: Error finding container edf751045eb8a4bed2bc125385f9cafa79ea0296b21fa97e315f6ba58070bb44: Status 404 returned error can't find the container with id edf751045eb8a4bed2bc125385f9cafa79ea0296b21fa97e315f6ba58070bb44
Jul 24 00:57:05.006631 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.006615    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:57:05.160773 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.160694    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"]
Jul 24 00:57:05.165039 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.165014    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:57:05.168386 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.168365    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-193ff5\""
Jul 24 00:57:05.168525 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.168429    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-193ff5-dockercfg-nvqtl\""
Jul 24 00:57:05.174482 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.174459    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"]
Jul 24 00:57:05.195185 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.195156    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"]
Jul 24 00:57:05.198274 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.198259    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:57:05.208664 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.208641    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"]
Jul 24 00:57:05.326440 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.326406    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941b00f1-fd05-4185-b45a-88848cae5210-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj\" (UID: \"941b00f1-fd05-4185-b45a-88848cae5210\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:57:05.326603 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.326452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58674b1-ba18-40f1-a58e-0cc79966a06c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn\" (UID: \"d58674b1-ba18-40f1-a58e-0cc79966a06c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:57:05.326603 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.326523    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbt7z\" (UniqueName: \"kubernetes.io/projected/d58674b1-ba18-40f1-a58e-0cc79966a06c-kube-api-access-lbt7z\") pod \"payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn\" (UID: \"d58674b1-ba18-40f1-a58e-0cc79966a06c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:57:05.326603 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.326563    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvqm4\" (UniqueName: \"kubernetes.io/projected/941b00f1-fd05-4185-b45a-88848cae5210-kube-api-access-lvqm4\") pod \"payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj\" (UID: \"941b00f1-fd05-4185-b45a-88848cae5210\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:57:05.427476 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.427446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbt7z\" (UniqueName: \"kubernetes.io/projected/d58674b1-ba18-40f1-a58e-0cc79966a06c-kube-api-access-lbt7z\") pod \"payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn\" (UID: \"d58674b1-ba18-40f1-a58e-0cc79966a06c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:57:05.427568 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.427481    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvqm4\" (UniqueName: \"kubernetes.io/projected/941b00f1-fd05-4185-b45a-88848cae5210-kube-api-access-lvqm4\") pod \"payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj\" (UID: \"941b00f1-fd05-4185-b45a-88848cae5210\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:57:05.427568 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.427527    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941b00f1-fd05-4185-b45a-88848cae5210-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj\" (UID: \"941b00f1-fd05-4185-b45a-88848cae5210\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:57:05.427568 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.427559    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58674b1-ba18-40f1-a58e-0cc79966a06c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn\" (UID: \"d58674b1-ba18-40f1-a58e-0cc79966a06c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:57:05.428211 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.428192    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58674b1-ba18-40f1-a58e-0cc79966a06c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn\" (UID: \"d58674b1-ba18-40f1-a58e-0cc79966a06c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:57:05.428252 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.428229    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941b00f1-fd05-4185-b45a-88848cae5210-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj\" (UID: \"941b00f1-fd05-4185-b45a-88848cae5210\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:57:05.435196 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.435171    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvqm4\" (UniqueName: \"kubernetes.io/projected/941b00f1-fd05-4185-b45a-88848cae5210-kube-api-access-lvqm4\") pod \"payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj\" (UID: \"941b00f1-fd05-4185-b45a-88848cae5210\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:57:05.435290 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.435214    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbt7z\" (UniqueName: \"kubernetes.io/projected/d58674b1-ba18-40f1-a58e-0cc79966a06c-kube-api-access-lbt7z\") pod \"payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn\" (UID: \"d58674b1-ba18-40f1-a58e-0cc79966a06c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:57:05.475775 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.475741    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:57:05.508180 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.508145    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:57:05.621601 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.621570    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"]
Jul 24 00:57:05.624925 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:57:05.624896    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941b00f1_fd05_4185_b45a_88848cae5210.slice/crio-8ef461a0b87e69516e7f98dabab9a8dc06bd5038d62d32fcbff455e0f23c0b97 WatchSource:0}: Error finding container 8ef461a0b87e69516e7f98dabab9a8dc06bd5038d62d32fcbff455e0f23c0b97: Status 404 returned error can't find the container with id 8ef461a0b87e69516e7f98dabab9a8dc06bd5038d62d32fcbff455e0f23c0b97
Jul 24 00:57:05.653095 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.653067    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"]
Jul 24 00:57:05.657428 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:57:05.657400    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58674b1_ba18_40f1_a58e_0cc79966a06c.slice/crio-3295f0bcb3349b867590c6393075108636b6013454a9ae5837d17a57d13d0e4e WatchSource:0}: Error finding container 3295f0bcb3349b867590c6393075108636b6013454a9ae5837d17a57d13d0e4e: Status 404 returned error can't find the container with id 3295f0bcb3349b867590c6393075108636b6013454a9ae5837d17a57d13d0e4e
Jul 24 00:57:05.987374 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.987267    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn" event={"ID":"d58674b1-ba18-40f1-a58e-0cc79966a06c","Type":"ContainerStarted","Data":"6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac"}
Jul 24 00:57:05.987374 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.987353    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn" event={"ID":"d58674b1-ba18-40f1-a58e-0cc79966a06c","Type":"ContainerStarted","Data":"3295f0bcb3349b867590c6393075108636b6013454a9ae5837d17a57d13d0e4e"}
Jul 24 00:57:05.987583 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.987384    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:57:05.988983 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.988950    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj" event={"ID":"941b00f1-fd05-4185-b45a-88848cae5210","Type":"ContainerStarted","Data":"885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33"}
Jul 24 00:57:05.989117 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.988997    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:57:05.989117 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.989020    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj" event={"ID":"941b00f1-fd05-4185-b45a-88848cae5210","Type":"ContainerStarted","Data":"8ef461a0b87e69516e7f98dabab9a8dc06bd5038d62d32fcbff455e0f23c0b97"}
Jul 24 00:57:05.990358 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.990331    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z" event={"ID":"e1793562-f014-47a1-9161-c3ae18439fde","Type":"ContainerStarted","Data":"bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5"}
Jul 24 00:57:05.990451 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.990377    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z" event={"ID":"e1793562-f014-47a1-9161-c3ae18439fde","Type":"ContainerStarted","Data":"edf751045eb8a4bed2bc125385f9cafa79ea0296b21fa97e315f6ba58070bb44"}
Jul 24 00:57:05.990451 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:05.990400    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:06.004785 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:06.004742    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn" podStartSLOduration=1.004727947 podStartE2EDuration="1.004727947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:57:06.003195952 +0000 UTC m=+2788.851514710" watchObservedRunningTime="2026-07-24 00:57:06.004727947 +0000 UTC m=+2788.853046688"
Jul 24 00:57:06.019163 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:06.019124    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z" podStartSLOduration=1.662978385 podStartE2EDuration="2.019111174s" totalImagesPullingTime="356.132789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:57:06.017330702 +0000 UTC m=+2788.865649438" watchObservedRunningTime="2026-07-24 00:57:06.019111174 +0000 UTC m=+2788.867429915"
Jul 24 00:57:06.030809 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:06.030769    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj" podStartSLOduration=1.030756747 podStartE2EDuration="1.030756747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:57:06.03036812 +0000 UTC m=+2788.878686861" watchObservedRunningTime="2026-07-24 00:57:06.030756747 +0000 UTC m=+2788.879075487"
Jul 24 00:57:12.005168 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:12.005138    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:57:17.002232 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:17.002200    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:57:17.002637 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:17.002268    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:57:20.263790 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.263748    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"]
Jul 24 00:57:20.266994 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.266974    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.269482 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.269458    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4fa1d5-openshift-default-dockercfg-8xllw\""
Jul 24 00:57:20.280143 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.280116    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"]
Jul 24 00:57:20.346849 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.346816    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-envoy\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.347032 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.346863    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4zjk\" (UniqueName: \"kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-kube-api-access-h4zjk\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.347032 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.346894    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-certs\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.347032 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.346947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-data\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.347032 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.346978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6af879-5324-4675-970d-83da0dd16b6e-istio-podinfo\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.347032 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.346998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6af879-5324-4675-970d-83da0dd16b6e-istiod-ca-cert\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.347032 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.347024    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-credential-socket\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.347238 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.347063    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-socket\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.347238 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.347113    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-istio-token\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.448398 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.448361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-istio-token\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.448585 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.448488    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-envoy\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.448585 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.448539    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zjk\" (UniqueName: \"kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-kube-api-access-h4zjk\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.448585 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.448572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-certs\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.448757 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.448601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-data\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.448757 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.448625    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6af879-5324-4675-970d-83da0dd16b6e-istio-podinfo\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.448757 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.448649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6af879-5324-4675-970d-83da0dd16b6e-istiod-ca-cert\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.448757 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.448679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-credential-socket\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.448757 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.448712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-socket\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.449190 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.449163    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-socket\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.449290 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.449188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-certs\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.449527 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.449508    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-data\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.449612 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.449592    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-credential-socket\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.449889 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.449868    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6af879-5324-4675-970d-83da0dd16b6e-istiod-ca-cert\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.451383 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.451359    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-envoy\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.451602 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.451582    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6af879-5324-4675-970d-83da0dd16b6e-istio-podinfo\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.456534 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.456511    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-istio-token\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.457365 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.457343    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4zjk\" (UniqueName: \"kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-kube-api-access-h4zjk\") pod \"e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") " pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.578009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.577979    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:20.717655 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.717631    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"]
Jul 24 00:57:20.719728 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:57:20.719693    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6af879_5324_4675_970d_83da0dd16b6e.slice/crio-b1964b7a72a17a7cb0526b7e48e35ce506b43f5d1443e1d5aac1c048512f49c2 WatchSource:0}: Error finding container b1964b7a72a17a7cb0526b7e48e35ce506b43f5d1443e1d5aac1c048512f49c2: Status 404 returned error can't find the container with id b1964b7a72a17a7cb0526b7e48e35ce506b43f5d1443e1d5aac1c048512f49c2
Jul 24 00:57:20.721993 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.721953    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:57:20.722073 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.722039    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:57:20.722117 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:20.722079    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:57:21.065371 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:21.065256    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" event={"ID":"5a6af879-5324-4675-970d-83da0dd16b6e","Type":"ContainerStarted","Data":"a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb"}
Jul 24 00:57:21.065371 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:21.065334    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" event={"ID":"5a6af879-5324-4675-970d-83da0dd16b6e","Type":"ContainerStarted","Data":"b1964b7a72a17a7cb0526b7e48e35ce506b43f5d1443e1d5aac1c048512f49c2"}
Jul 24 00:57:21.084650 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:21.084596    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" podStartSLOduration=1.084580555 podStartE2EDuration="1.084580555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:57:21.083512048 +0000 UTC m=+2803.931830786" watchObservedRunningTime="2026-07-24 00:57:21.084580555 +0000 UTC m=+2803.932899295"
Jul 24 00:57:21.578146 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:21.578107    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:21.584375 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:21.584344    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:22.070143 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:22.070105    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:22.071054 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:22.071031    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:40.148548 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:40.148466    2574 generic.go:356] "Generic (PLEG): container finished" podID="5a6af879-5324-4675-970d-83da0dd16b6e" containerID="a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb" exitCode=0
Jul 24 00:57:40.148548 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:40.148526    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" event={"ID":"5a6af879-5324-4675-970d-83da0dd16b6e","Type":"ContainerDied","Data":"a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb"}
Jul 24 00:57:40.148960 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:40.148853    2574 scope.go:117] "RemoveContainer" containerID="a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb"
Jul 24 00:57:40.150009 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:40.149980    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:57:40.150076 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:40.150055    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:57:40.150111 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:40.150093    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 00:57:41.157258 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:41.157212    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" event={"ID":"5a6af879-5324-4675-970d-83da0dd16b6e","Type":"ContainerStarted","Data":"394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959"}
Jul 24 00:57:41.578902 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:41.578867    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:41.579175 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:41.579153    2574 patch_prober.go:28] interesting pod/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 00:57:41.579233 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:41.579214    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 00:57:42.578641 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:42.578606    2574 patch_prober.go:28] interesting pod/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 00:57:42.579067 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:42.578667    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 00:57:43.578602 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:43.578566    2574 patch_prober.go:28] interesting pod/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 00:57:43.578871 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:43.578624    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 00:57:44.579165 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:44.579126    2574 patch_prober.go:28] interesting pod/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 00:57:44.579582 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:44.579183    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 00:57:45.582006 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:45.581977    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:45.582392 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:45.582340    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:45.582949 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:45.582930    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 00:57:53.738792 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.738761    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"]
Jul 24 00:57:53.742122 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.742105    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.745743 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.745718    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-193ff5\"/\"default-dockercfg-vlv46\""
Jul 24 00:57:53.745840 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.745787    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-193ff5\"/\"ipp-route-193ff5-kserve-self-signed-certs\""
Jul 24 00:57:53.745840 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.745817    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-193ff5\"/\"kube-root-ca.crt\""
Jul 24 00:57:53.745943 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.745860    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-193ff5\"/\"openshift-service-ca.crt\""
Jul 24 00:57:53.755025 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.754997    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"]
Jul 24 00:57:53.843869 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.843841    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tls-certs\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.844030 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.843877    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-home\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.844030 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.843899    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-dshm\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.844030 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.843925    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tmp-dir\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.844030 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.843980    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-model-cache\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.844174 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.844084    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6c4d\" (UniqueName: \"kubernetes.io/projected/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-kube-api-access-p6c4d\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.945257 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.945221    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6c4d\" (UniqueName: \"kubernetes.io/projected/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-kube-api-access-p6c4d\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.945449 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.945340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tls-certs\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.945449 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.945387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-home\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.945449 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.945428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-dshm\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.945616 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.945478    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tmp-dir\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.945616 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.945507    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-model-cache\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.945787 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.945769    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-home\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.945855 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.945833    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tmp-dir\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.945933 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.945913    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-model-cache\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.947683 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.947655    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-dshm\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.947802 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.947785    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tls-certs\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:53.953596 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:53.953571    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6c4d\" (UniqueName: \"kubernetes.io/projected/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-kube-api-access-p6c4d\") pod \"ipp-route-193ff5-kserve-5b57c784f5-62655\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") " pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:54.051584 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:54.051523    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:57:54.177532 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:54.177509    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"]
Jul 24 00:57:54.179298 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:57:54.179268    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe8c1d66_20fb_4ff9_b49e_1cd5dfa86676.slice/crio-ed95d1ab254f4a30c542833dbc9776ebbacdc714fa7dcba5bafa96c34b9e474b WatchSource:0}: Error finding container ed95d1ab254f4a30c542833dbc9776ebbacdc714fa7dcba5bafa96c34b9e474b: Status 404 returned error can't find the container with id ed95d1ab254f4a30c542833dbc9776ebbacdc714fa7dcba5bafa96c34b9e474b
Jul 24 00:57:54.214156 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:54.214131    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655" event={"ID":"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676","Type":"ContainerStarted","Data":"ed95d1ab254f4a30c542833dbc9776ebbacdc714fa7dcba5bafa96c34b9e474b"}
Jul 24 00:57:55.219720 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:55.219682    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655" event={"ID":"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676","Type":"ContainerStarted","Data":"e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba"}
Jul 24 00:57:55.243424 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:57:55.243376    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655" podStartSLOduration=2.243360306 podStartE2EDuration="2.243360306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:57:55.241636457 +0000 UTC m=+2838.089955199" watchObservedRunningTime="2026-07-24 00:57:55.243360306 +0000 UTC m=+2838.091679047"
Jul 24 00:58:04.051682 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:04.051647    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:58:04.052083 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:04.051702    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:58:04.064233 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:04.064206    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:58:04.270724 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:04.270690    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:58:31.888529 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:31.888419    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn_d58674b1-ba18-40f1-a58e-0cc79966a06c/payload-processing/0.log"
Jul 24 00:58:36.739328 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:36.739283    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"]
Jul 24 00:58:36.742717 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:36.742701    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:58:36.745484 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:36.745462    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-78d793-dockercfg-2sccc\""
Jul 24 00:58:36.745484 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:36.745477    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-78d793\""
Jul 24 00:58:36.752035 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:36.752009    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"]
Jul 24 00:58:36.825280 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:36.825247    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c88hv\" (UniqueName: \"kubernetes.io/projected/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-kube-api-access-c88hv\") pod \"payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49\" (UID: \"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:58:36.825494 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:36.825374    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49\" (UID: \"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:58:36.926580 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:36.926552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c88hv\" (UniqueName: \"kubernetes.io/projected/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-kube-api-access-c88hv\") pod \"payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49\" (UID: \"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:58:36.926739 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:36.926615    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49\" (UID: \"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:58:36.927291 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:36.927271    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49\" (UID: \"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:58:36.935097 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:36.935067    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c88hv\" (UniqueName: \"kubernetes.io/projected/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-kube-api-access-c88hv\") pod \"payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49\" (UID: \"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:58:37.054247 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:37.054176    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:58:37.182589 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:37.182564    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"]
Jul 24 00:58:37.184753 ip-10-0-139-9 kubenswrapper[2574]: W0724 00:58:37.184720    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4dbbb6c_5bef_44fb_b86c_1d14c425c3c3.slice/crio-791ba9dae55360e17859452637b92a71c0b73e037d0ef28b6b1eb1d0803c7024 WatchSource:0}: Error finding container 791ba9dae55360e17859452637b92a71c0b73e037d0ef28b6b1eb1d0803c7024: Status 404 returned error can't find the container with id 791ba9dae55360e17859452637b92a71c0b73e037d0ef28b6b1eb1d0803c7024
Jul 24 00:58:37.411401 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:37.411365    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49" event={"ID":"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3","Type":"ContainerStarted","Data":"f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65"}
Jul 24 00:58:37.411590 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:37.411429    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49" event={"ID":"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3","Type":"ContainerStarted","Data":"791ba9dae55360e17859452637b92a71c0b73e037d0ef28b6b1eb1d0803c7024"}
Jul 24 00:58:37.411590 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:37.411509    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:58:37.427450 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:37.427404    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49" podStartSLOduration=1.427391396 podStartE2EDuration="1.427391396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:58:37.426071024 +0000 UTC m=+2880.274389769" watchObservedRunningTime="2026-07-24 00:58:37.427391396 +0000 UTC m=+2880.275710137"
Jul 24 00:58:48.418273 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:58:48.418243    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:59:02.018219 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.018186    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"]
Jul 24 00:59:02.018857 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.018461    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49" podUID="d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3" containerName="payload-processing" containerID="cri-o://f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65" gracePeriod=30
Jul 24 00:59:02.281829 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.281765    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:59:02.347930 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.347890    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c88hv\" (UniqueName: \"kubernetes.io/projected/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-kube-api-access-c88hv\") pod \"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3\" (UID: \"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3\") "
Jul 24 00:59:02.348121 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.347964    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-plugins-config-volume\") pod \"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3\" (UID: \"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3\") "
Jul 24 00:59:02.348391 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.348366    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3" (UID: "d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:59:02.350066 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.350037    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-kube-api-access-c88hv" (OuterVolumeSpecName: "kube-api-access-c88hv") pod "d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3" (UID: "d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3"). InnerVolumeSpecName "kube-api-access-c88hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:59:02.448950 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.448900    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c88hv\" (UniqueName: \"kubernetes.io/projected/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-kube-api-access-c88hv\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:02.448950 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.448944    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3-plugins-config-volume\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:02.526855 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.526821    2574 generic.go:356] "Generic (PLEG): container finished" podID="d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3" containerID="f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65" exitCode=0
Jul 24 00:59:02.527049 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.526865    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"
Jul 24 00:59:02.527049 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.526870    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49" event={"ID":"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3","Type":"ContainerDied","Data":"f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65"}
Jul 24 00:59:02.527049 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.527000    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49" event={"ID":"d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3","Type":"ContainerDied","Data":"791ba9dae55360e17859452637b92a71c0b73e037d0ef28b6b1eb1d0803c7024"}
Jul 24 00:59:02.527049 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.527042    2574 scope.go:117] "RemoveContainer" containerID="f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65"
Jul 24 00:59:02.545799 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.545767    2574 scope.go:117] "RemoveContainer" containerID="f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65"
Jul 24 00:59:02.546338 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:59:02.546292    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65\": container with ID starting with f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65 not found: ID does not exist" containerID="f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65"
Jul 24 00:59:02.546434 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.546350    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65"} err="failed to get container status \"f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65\": rpc error: code = NotFound desc = could not find container \"f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65\": container with ID starting with f52e9f77743fae8daa3bac60937e15815ed6846faad8f76325eb79f47c753d65 not found: ID does not exist"
Jul 24 00:59:02.556514 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.556485    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"]
Jul 24 00:59:02.558518 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:02.558497    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-78d793-58658585b8-z8l49"]
Jul 24 00:59:03.752118 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:03.752082    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3" path="/var/lib/kubelet/pods/d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3/volumes"
Jul 24 00:59:32.177581 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.177545    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"]
Jul 24 00:59:32.178718 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.177763    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z" podUID="e1793562-f014-47a1-9161-c3ae18439fde" containerName="maas-api" containerID="cri-o://bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5" gracePeriod=30
Jul 24 00:59:32.234128 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.234090    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"]
Jul 24 00:59:32.234382 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.234355    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn" podUID="d58674b1-ba18-40f1-a58e-0cc79966a06c" containerName="payload-processing" containerID="cri-o://6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac" gracePeriod=30
Jul 24 00:59:32.240519 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.240489    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"]
Jul 24 00:59:32.240736 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.240712    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj" podUID="941b00f1-fd05-4185-b45a-88848cae5210" containerName="payload-pre-processing" containerID="cri-o://885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33" gracePeriod=30
Jul 24 00:59:32.527753 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.527725    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:59:32.530407 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.530385    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:59:32.549246 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.549151    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:59:32.611650 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.611614    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58674b1-ba18-40f1-a58e-0cc79966a06c-plugins-config-volume\") pod \"d58674b1-ba18-40f1-a58e-0cc79966a06c\" (UID: \"d58674b1-ba18-40f1-a58e-0cc79966a06c\") "
Jul 24 00:59:32.611847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.611661    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1793562-f014-47a1-9161-c3ae18439fde-maas-api-tls\") pod \"e1793562-f014-47a1-9161-c3ae18439fde\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") "
Jul 24 00:59:32.611847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.611686    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqrwz\" (UniqueName: \"kubernetes.io/projected/e1793562-f014-47a1-9161-c3ae18439fde-kube-api-access-vqrwz\") pod \"e1793562-f014-47a1-9161-c3ae18439fde\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") "
Jul 24 00:59:32.611847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.611720    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvqm4\" (UniqueName: \"kubernetes.io/projected/941b00f1-fd05-4185-b45a-88848cae5210-kube-api-access-lvqm4\") pod \"941b00f1-fd05-4185-b45a-88848cae5210\" (UID: \"941b00f1-fd05-4185-b45a-88848cae5210\") "
Jul 24 00:59:32.611847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.611741    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1793562-f014-47a1-9161-c3ae18439fde-openshift-service-ca\") pod \"e1793562-f014-47a1-9161-c3ae18439fde\" (UID: \"e1793562-f014-47a1-9161-c3ae18439fde\") "
Jul 24 00:59:32.611847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.611775    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbt7z\" (UniqueName: \"kubernetes.io/projected/d58674b1-ba18-40f1-a58e-0cc79966a06c-kube-api-access-lbt7z\") pod \"d58674b1-ba18-40f1-a58e-0cc79966a06c\" (UID: \"d58674b1-ba18-40f1-a58e-0cc79966a06c\") "
Jul 24 00:59:32.611847 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.611815    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941b00f1-fd05-4185-b45a-88848cae5210-plugins-config-volume\") pod \"941b00f1-fd05-4185-b45a-88848cae5210\" (UID: \"941b00f1-fd05-4185-b45a-88848cae5210\") "
Jul 24 00:59:32.612171 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.612092    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58674b1-ba18-40f1-a58e-0cc79966a06c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d58674b1-ba18-40f1-a58e-0cc79966a06c" (UID: "d58674b1-ba18-40f1-a58e-0cc79966a06c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:59:32.612171 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.612152    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1793562-f014-47a1-9161-c3ae18439fde-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1793562-f014-47a1-9161-c3ae18439fde" (UID: "e1793562-f014-47a1-9161-c3ae18439fde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:59:32.612496 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.612452    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941b00f1-fd05-4185-b45a-88848cae5210-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "941b00f1-fd05-4185-b45a-88848cae5210" (UID: "941b00f1-fd05-4185-b45a-88848cae5210"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:59:32.613943 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.613916    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58674b1-ba18-40f1-a58e-0cc79966a06c-kube-api-access-lbt7z" (OuterVolumeSpecName: "kube-api-access-lbt7z") pod "d58674b1-ba18-40f1-a58e-0cc79966a06c" (UID: "d58674b1-ba18-40f1-a58e-0cc79966a06c"). InnerVolumeSpecName "kube-api-access-lbt7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:59:32.614045 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.613952    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1793562-f014-47a1-9161-c3ae18439fde-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1793562-f014-47a1-9161-c3ae18439fde" (UID: "e1793562-f014-47a1-9161-c3ae18439fde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:59:32.614045 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.613957    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941b00f1-fd05-4185-b45a-88848cae5210-kube-api-access-lvqm4" (OuterVolumeSpecName: "kube-api-access-lvqm4") pod "941b00f1-fd05-4185-b45a-88848cae5210" (UID: "941b00f1-fd05-4185-b45a-88848cae5210"). InnerVolumeSpecName "kube-api-access-lvqm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:59:32.614045 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.613975    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1793562-f014-47a1-9161-c3ae18439fde-kube-api-access-vqrwz" (OuterVolumeSpecName: "kube-api-access-vqrwz") pod "e1793562-f014-47a1-9161-c3ae18439fde" (UID: "e1793562-f014-47a1-9161-c3ae18439fde"). InnerVolumeSpecName "kube-api-access-vqrwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:59:32.660621 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.660587    2574 generic.go:356] "Generic (PLEG): container finished" podID="d58674b1-ba18-40f1-a58e-0cc79966a06c" containerID="6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac" exitCode=0
Jul 24 00:59:32.660800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.660636    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"
Jul 24 00:59:32.660800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.660680    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn" event={"ID":"d58674b1-ba18-40f1-a58e-0cc79966a06c","Type":"ContainerDied","Data":"6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac"}
Jul 24 00:59:32.660800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.660739    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn" event={"ID":"d58674b1-ba18-40f1-a58e-0cc79966a06c","Type":"ContainerDied","Data":"3295f0bcb3349b867590c6393075108636b6013454a9ae5837d17a57d13d0e4e"}
Jul 24 00:59:32.660800 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.660768    2574 scope.go:117] "RemoveContainer" containerID="6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac"
Jul 24 00:59:32.662044 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.662024    2574 generic.go:356] "Generic (PLEG): container finished" podID="941b00f1-fd05-4185-b45a-88848cae5210" containerID="885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33" exitCode=0
Jul 24 00:59:32.662158 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.662072    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"
Jul 24 00:59:32.662158 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.662098    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj" event={"ID":"941b00f1-fd05-4185-b45a-88848cae5210","Type":"ContainerDied","Data":"885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33"}
Jul 24 00:59:32.662158 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.662140    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj" event={"ID":"941b00f1-fd05-4185-b45a-88848cae5210","Type":"ContainerDied","Data":"8ef461a0b87e69516e7f98dabab9a8dc06bd5038d62d32fcbff455e0f23c0b97"}
Jul 24 00:59:32.666580 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.664150    2574 generic.go:356] "Generic (PLEG): container finished" podID="e1793562-f014-47a1-9161-c3ae18439fde" containerID="bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5" exitCode=0
Jul 24 00:59:32.666580 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.664509    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"
Jul 24 00:59:32.666580 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.665648    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z" event={"ID":"e1793562-f014-47a1-9161-c3ae18439fde","Type":"ContainerDied","Data":"bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5"}
Jul 24 00:59:32.666580 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.665732    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z" event={"ID":"e1793562-f014-47a1-9161-c3ae18439fde","Type":"ContainerDied","Data":"edf751045eb8a4bed2bc125385f9cafa79ea0296b21fa97e315f6ba58070bb44"}
Jul 24 00:59:32.678452 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.678432    2574 scope.go:117] "RemoveContainer" containerID="6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac"
Jul 24 00:59:32.679044 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:59:32.679011    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac\": container with ID starting with 6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac not found: ID does not exist" containerID="6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac"
Jul 24 00:59:32.679185 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.679162    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac"} err="failed to get container status \"6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac\": rpc error: code = NotFound desc = could not find container \"6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac\": container with ID starting with 6faaa07f4403666c4d31d7f45c6906ea447d9d0bb707bbae1a1afe19a9ea54ac not found: ID does not exist"
Jul 24 00:59:32.679268 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.679258    2574 scope.go:117] "RemoveContainer" containerID="885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33"
Jul 24 00:59:32.687622 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.687604    2574 scope.go:117] "RemoveContainer" containerID="885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33"
Jul 24 00:59:32.687887 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:59:32.687867    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33\": container with ID starting with 885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33 not found: ID does not exist" containerID="885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33"
Jul 24 00:59:32.687947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.687893    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33"} err="failed to get container status \"885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33\": rpc error: code = NotFound desc = could not find container \"885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33\": container with ID starting with 885ddcc5b3353deab1df6081db5724b4ef3319e5270891c8939a5ae71191ab33 not found: ID does not exist"
Jul 24 00:59:32.687947 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.687909    2574 scope.go:117] "RemoveContainer" containerID="bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5"
Jul 24 00:59:32.693358 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.693337    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"]
Jul 24 00:59:32.695840 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.695821    2574 scope.go:117] "RemoveContainer" containerID="bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5"
Jul 24 00:59:32.696088 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:59:32.696070    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5\": container with ID starting with bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5 not found: ID does not exist" containerID="bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5"
Jul 24 00:59:32.696143 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.696094    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5"} err="failed to get container status \"bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5\": rpc error: code = NotFound desc = could not find container \"bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5\": container with ID starting with bc86f0f70cacc9958bfc736fe9d3562326ec9c05dad4ea9adeb60de24b0094c5 not found: ID does not exist"
Jul 24 00:59:32.696864 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.696847    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-193ff5-6f4ddd5f7d-x22kn"]
Jul 24 00:59:32.706845 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.706822    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"]
Jul 24 00:59:32.712034 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.712016    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-193ff5-f48596b76-fkjtj"]
Jul 24 00:59:32.712930 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.712912    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbt7z\" (UniqueName: \"kubernetes.io/projected/d58674b1-ba18-40f1-a58e-0cc79966a06c-kube-api-access-lbt7z\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:32.712990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.712933    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941b00f1-fd05-4185-b45a-88848cae5210-plugins-config-volume\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:32.712990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.712944    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58674b1-ba18-40f1-a58e-0cc79966a06c-plugins-config-volume\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:32.712990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.712953    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1793562-f014-47a1-9161-c3ae18439fde-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:32.712990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.712962    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqrwz\" (UniqueName: \"kubernetes.io/projected/e1793562-f014-47a1-9161-c3ae18439fde-kube-api-access-vqrwz\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:32.712990 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.712970    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvqm4\" (UniqueName: \"kubernetes.io/projected/941b00f1-fd05-4185-b45a-88848cae5210-kube-api-access-lvqm4\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:32.713150 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.712997    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1793562-f014-47a1-9161-c3ae18439fde-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:32.722065 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.722044    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"]
Jul 24 00:59:32.725861 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:32.725840    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-193ff5-6dbd6df668-6nb2z"]
Jul 24 00:59:33.752323 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:33.752277    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="941b00f1-fd05-4185-b45a-88848cae5210" path="/var/lib/kubelet/pods/941b00f1-fd05-4185-b45a-88848cae5210/volumes"
Jul 24 00:59:33.752739 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:33.752672    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d58674b1-ba18-40f1-a58e-0cc79966a06c" path="/var/lib/kubelet/pods/d58674b1-ba18-40f1-a58e-0cc79966a06c/volumes"
Jul 24 00:59:33.753012 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:33.752999    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1793562-f014-47a1-9161-c3ae18439fde" path="/var/lib/kubelet/pods/e1793562-f014-47a1-9161-c3ae18439fde/volumes"
Jul 24 00:59:42.082479 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:42.082440    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"]
Jul 24 00:59:57.100911 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.100856    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655" podUID="fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" containerName="main" containerID="cri-o://e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba" gracePeriod=45
Jul 24 00:59:57.336600 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.336576    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:59:57.419024 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.418939    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tls-certs\") pod \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") "
Jul 24 00:59:57.419024 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.418984    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6c4d\" (UniqueName: \"kubernetes.io/projected/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-kube-api-access-p6c4d\") pod \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") "
Jul 24 00:59:57.419209 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.419060    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tmp-dir\") pod \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") "
Jul 24 00:59:57.419209 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.419096    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-home\") pod \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") "
Jul 24 00:59:57.419209 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.419133    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-dshm\") pod \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") "
Jul 24 00:59:57.419209 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.419187    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-model-cache\") pod \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\" (UID: \"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676\") "
Jul 24 00:59:57.419439 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.419348    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" (UID: "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:59:57.419439 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.419367    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-home" (OuterVolumeSpecName: "home") pod "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" (UID: "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:59:57.419552 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.419531    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" (UID: "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:59:57.419608 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.419563    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tmp-dir\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:57.419608 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.419583    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-home\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:57.421180 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.421145    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-dshm" (OuterVolumeSpecName: "dshm") pod "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" (UID: "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:59:57.421301 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.421215    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" (UID: "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:59:57.421301 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.421226    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-kube-api-access-p6c4d" (OuterVolumeSpecName: "kube-api-access-p6c4d") pod "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" (UID: "fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676"). InnerVolumeSpecName "kube-api-access-p6c4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:59:57.520831 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.520802    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-dshm\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:57.520831 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.520828    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-model-cache\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:57.521001 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.520839    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-tls-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:57.521001 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.520848    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6c4d\" (UniqueName: \"kubernetes.io/projected/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676-kube-api-access-p6c4d\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 00:59:57.776210 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.776182    2574 generic.go:356] "Generic (PLEG): container finished" podID="fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" containerID="e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba" exitCode=0
Jul 24 00:59:57.776353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.776225    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"
Jul 24 00:59:57.776353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.776268    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655" event={"ID":"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676","Type":"ContainerDied","Data":"e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba"}
Jul 24 00:59:57.776353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.776333    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655" event={"ID":"fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676","Type":"ContainerDied","Data":"ed95d1ab254f4a30c542833dbc9776ebbacdc714fa7dcba5bafa96c34b9e474b"}
Jul 24 00:59:57.776353 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.776344    2574 scope.go:117] "RemoveContainer" containerID="e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba"
Jul 24 00:59:57.784213 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.784203    2574 scope.go:117] "RemoveContainer" containerID="e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba"
Jul 24 00:59:57.784452 ip-10-0-139-9 kubenswrapper[2574]: E0724 00:59:57.784435    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba\": container with ID starting with e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba not found: ID does not exist" containerID="e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba"
Jul 24 00:59:57.784508 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.784459    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba"} err="failed to get container status \"e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba\": rpc error: code = NotFound desc = could not find container \"e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba\": container with ID starting with e91b279bffa92f7a08536ddcddb7efa64fb8d8c452655625a482d4a6188319ba not found: ID does not exist"
Jul 24 00:59:57.791899 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.791878    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"]
Jul 24 00:59:57.795893 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:57.795873    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-193ff5/ipp-route-193ff5-kserve-5b57c784f5-62655"]
Jul 24 00:59:59.751458 ip-10-0-139-9 kubenswrapper[2574]: I0724 00:59:59.751363    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" path="/var/lib/kubelet/pods/fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676/volumes"
Jul 24 01:00:47.113091 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:47.113057    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"]
Jul 24 01:00:47.113606 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:47.113331    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" containerName="istio-proxy" containerID="cri-o://394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959" gracePeriod=30
Jul 24 01:00:52.369130 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.369086    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 01:00:52.510157 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510062    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-data\") pod \"5a6af879-5324-4675-970d-83da0dd16b6e\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") "
Jul 24 01:00:52.510157 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510102    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-socket\") pod \"5a6af879-5324-4675-970d-83da0dd16b6e\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") "
Jul 24 01:00:52.510386 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510160    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-credential-socket\") pod \"5a6af879-5324-4675-970d-83da0dd16b6e\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") "
Jul 24 01:00:52.510386 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510179    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-certs\") pod \"5a6af879-5324-4675-970d-83da0dd16b6e\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") "
Jul 24 01:00:52.510386 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510209    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-envoy\") pod \"5a6af879-5324-4675-970d-83da0dd16b6e\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") "
Jul 24 01:00:52.510386 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510256    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-istio-token\") pod \"5a6af879-5324-4675-970d-83da0dd16b6e\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") "
Jul 24 01:00:52.510386 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510358    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6af879-5324-4675-970d-83da0dd16b6e-istio-podinfo\") pod \"5a6af879-5324-4675-970d-83da0dd16b6e\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") "
Jul 24 01:00:52.510647 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510400    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4zjk\" (UniqueName: \"kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-kube-api-access-h4zjk\") pod \"5a6af879-5324-4675-970d-83da0dd16b6e\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") "
Jul 24 01:00:52.510647 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510439    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6af879-5324-4675-970d-83da0dd16b6e-istiod-ca-cert\") pod \"5a6af879-5324-4675-970d-83da0dd16b6e\" (UID: \"5a6af879-5324-4675-970d-83da0dd16b6e\") "
Jul 24 01:00:52.510647 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510502    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a6af879-5324-4675-970d-83da0dd16b6e" (UID: "5a6af879-5324-4675-970d-83da0dd16b6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:00:52.510647 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510532    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a6af879-5324-4675-970d-83da0dd16b6e" (UID: "5a6af879-5324-4675-970d-83da0dd16b6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:00:52.510866 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510635    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a6af879-5324-4675-970d-83da0dd16b6e" (UID: "5a6af879-5324-4675-970d-83da0dd16b6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:00:52.510866 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510767    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6af879-5324-4675-970d-83da0dd16b6e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a6af879-5324-4675-970d-83da0dd16b6e" (UID: "5a6af879-5324-4675-970d-83da0dd16b6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:00:52.510866 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510776    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:52.510866 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510793    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-credential-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:52.510866 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510807    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-workload-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:52.511116 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.510982    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a6af879-5324-4675-970d-83da0dd16b6e" (UID: "5a6af879-5324-4675-970d-83da0dd16b6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:00:52.512594 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.512561    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a6af879-5324-4675-970d-83da0dd16b6e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a6af879-5324-4675-970d-83da0dd16b6e" (UID: "5a6af879-5324-4675-970d-83da0dd16b6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 01:00:52.512720 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.512620    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a6af879-5324-4675-970d-83da0dd16b6e" (UID: "5a6af879-5324-4675-970d-83da0dd16b6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:00:52.512983 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.512961    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-kube-api-access-h4zjk" (OuterVolumeSpecName: "kube-api-access-h4zjk") pod "5a6af879-5324-4675-970d-83da0dd16b6e" (UID: "5a6af879-5324-4675-970d-83da0dd16b6e"). InnerVolumeSpecName "kube-api-access-h4zjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:00:52.512983 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.512964    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a6af879-5324-4675-970d-83da0dd16b6e" (UID: "5a6af879-5324-4675-970d-83da0dd16b6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:00:52.611426 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.611386    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-istio-token\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:52.611426 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.611418    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6af879-5324-4675-970d-83da0dd16b6e-istio-podinfo\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:52.611426 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.611428    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4zjk\" (UniqueName: \"kubernetes.io/projected/5a6af879-5324-4675-970d-83da0dd16b6e-kube-api-access-h4zjk\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:52.611426 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.611437    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6af879-5324-4675-970d-83da0dd16b6e-istiod-ca-cert\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:52.611693 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.611446    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-data\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:52.611693 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:52.611453    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6af879-5324-4675-970d-83da0dd16b6e-istio-envoy\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:53.005894 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.005856    2574 generic.go:356] "Generic (PLEG): container finished" podID="5a6af879-5324-4675-970d-83da0dd16b6e" containerID="394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959" exitCode=0
Jul 24 01:00:53.006067 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.005911    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" event={"ID":"5a6af879-5324-4675-970d-83da0dd16b6e","Type":"ContainerDied","Data":"394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959"}
Jul 24 01:00:53.006067 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.005913    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"
Jul 24 01:00:53.006067 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.005957    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt" event={"ID":"5a6af879-5324-4675-970d-83da0dd16b6e","Type":"ContainerDied","Data":"b1964b7a72a17a7cb0526b7e48e35ce506b43f5d1443e1d5aac1c048512f49c2"}
Jul 24 01:00:53.006067 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.005980    2574 scope.go:117] "RemoveContainer" containerID="394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959"
Jul 24 01:00:53.014632 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.014611    2574 scope.go:117] "RemoveContainer" containerID="a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb"
Jul 24 01:00:53.021837 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.021818    2574 scope.go:117] "RemoveContainer" containerID="394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959"
Jul 24 01:00:53.022106 ip-10-0-139-9 kubenswrapper[2574]: E0724 01:00:53.022084    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959\": container with ID starting with 394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959 not found: ID does not exist" containerID="394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959"
Jul 24 01:00:53.022156 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.022117    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959"} err="failed to get container status \"394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959\": rpc error: code = NotFound desc = could not find container \"394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959\": container with ID starting with 394dbb4fb4e03738835fa12ca5e9a7056d05493560228a94ea6e42af6820f959 not found: ID does not exist"
Jul 24 01:00:53.022156 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.022135    2574 scope.go:117] "RemoveContainer" containerID="a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb"
Jul 24 01:00:53.022375 ip-10-0-139-9 kubenswrapper[2574]: E0724 01:00:53.022357    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb\": container with ID starting with a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb not found: ID does not exist" containerID="a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb"
Jul 24 01:00:53.022443 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.022385    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb"} err="failed to get container status \"a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb\": rpc error: code = NotFound desc = could not find container \"a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb\": container with ID starting with a9efafbd92a5d1e7343ff4444c3f9a4999e39042a6555d3b87a61f714145a4fb not found: ID does not exist"
Jul 24 01:00:53.027066 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.027045    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"]
Jul 24 01:00:53.032899 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.032877    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fa1d5-openshift-default-d5594d9b6-b48wt"]
Jul 24 01:00:53.752235 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:53.752202    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" path="/var/lib/kubelet/pods/5a6af879-5324-4675-970d-83da0dd16b6e/volumes"
Jul 24 01:00:58.033986 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.033954    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"]
Jul 24 01:00:58.034491 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.034179    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz" podUID="84641ab4-14d6-4523-8aa8-5b865207d6c5" containerName="maas-api" containerID="cri-o://e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0" gracePeriod=30
Jul 24 01:00:58.277016 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.276983    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 01:00:58.353716 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.353685    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84641ab4-14d6-4523-8aa8-5b865207d6c5-openshift-service-ca\") pod \"84641ab4-14d6-4523-8aa8-5b865207d6c5\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") "
Jul 24 01:00:58.353716 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.353721    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5kvx\" (UniqueName: \"kubernetes.io/projected/84641ab4-14d6-4523-8aa8-5b865207d6c5-kube-api-access-v5kvx\") pod \"84641ab4-14d6-4523-8aa8-5b865207d6c5\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") "
Jul 24 01:00:58.353952 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.353748    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84641ab4-14d6-4523-8aa8-5b865207d6c5-maas-api-tls\") pod \"84641ab4-14d6-4523-8aa8-5b865207d6c5\" (UID: \"84641ab4-14d6-4523-8aa8-5b865207d6c5\") "
Jul 24 01:00:58.354115 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.354087    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84641ab4-14d6-4523-8aa8-5b865207d6c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84641ab4-14d6-4523-8aa8-5b865207d6c5" (UID: "84641ab4-14d6-4523-8aa8-5b865207d6c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:00:58.355872 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.355851    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84641ab4-14d6-4523-8aa8-5b865207d6c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84641ab4-14d6-4523-8aa8-5b865207d6c5" (UID: "84641ab4-14d6-4523-8aa8-5b865207d6c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:00:58.355955 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.355851    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84641ab4-14d6-4523-8aa8-5b865207d6c5-kube-api-access-v5kvx" (OuterVolumeSpecName: "kube-api-access-v5kvx") pod "84641ab4-14d6-4523-8aa8-5b865207d6c5" (UID: "84641ab4-14d6-4523-8aa8-5b865207d6c5"). InnerVolumeSpecName "kube-api-access-v5kvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:00:58.455254 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.455212    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84641ab4-14d6-4523-8aa8-5b865207d6c5-openshift-service-ca\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:58.455254 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.455245    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5kvx\" (UniqueName: \"kubernetes.io/projected/84641ab4-14d6-4523-8aa8-5b865207d6c5-kube-api-access-v5kvx\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:58.455254 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:58.455257    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84641ab4-14d6-4523-8aa8-5b865207d6c5-maas-api-tls\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:00:59.035091 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:59.035054    2574 generic.go:356] "Generic (PLEG): container finished" podID="84641ab4-14d6-4523-8aa8-5b865207d6c5" containerID="e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0" exitCode=0
Jul 24 01:00:59.035500 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:59.035104    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"
Jul 24 01:00:59.035500 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:59.035151    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz" event={"ID":"84641ab4-14d6-4523-8aa8-5b865207d6c5","Type":"ContainerDied","Data":"e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0"}
Jul 24 01:00:59.035500 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:59.035212    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz" event={"ID":"84641ab4-14d6-4523-8aa8-5b865207d6c5","Type":"ContainerDied","Data":"7b9c166a76f5126d4c4ce888e67ab2ec3c21e8449b21bb26e414917e7a81c447"}
Jul 24 01:00:59.035500 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:59.035246    2574 scope.go:117] "RemoveContainer" containerID="e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0"
Jul 24 01:00:59.043728 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:59.043710    2574 scope.go:117] "RemoveContainer" containerID="e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0"
Jul 24 01:00:59.043998 ip-10-0-139-9 kubenswrapper[2574]: E0724 01:00:59.043979    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0\": container with ID starting with e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0 not found: ID does not exist" containerID="e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0"
Jul 24 01:00:59.044067 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:59.044004    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0"} err="failed to get container status \"e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0\": rpc error: code = NotFound desc = could not find container \"e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0\": container with ID starting with e9b8c31ccccfbf90084cfe4641684a2bfb3b39793ad05ce1d51ef24cff32f7c0 not found: ID does not exist"
Jul 24 01:00:59.058815 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:59.058781    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"]
Jul 24 01:00:59.063187 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:59.063160    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz"]
Jul 24 01:00:59.752488 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:00:59.752447    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84641ab4-14d6-4523-8aa8-5b865207d6c5" path="/var/lib/kubelet/pods/84641ab4-14d6-4523-8aa8-5b865207d6c5/volumes"
Jul 24 01:01:01.247416 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:01.247359    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-44952a-6b4dbc644c-fp5vz" podUID="84641ab4-14d6-4523-8aa8-5b865207d6c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": context deadline exceeded"
Jul 24 01:01:08.949929 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:08.949894    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"]
Jul 24 01:01:08.950424 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:08.950092    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy" containerID="cri-o://7153c017d59862ad32ffeaeb82add09c16db6ef44844386ad31072f8bf19e72c" gracePeriod=30
Jul 24 01:01:14.100914 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.100878    2574 generic.go:356] "Generic (PLEG): container finished" podID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerID="7153c017d59862ad32ffeaeb82add09c16db6ef44844386ad31072f8bf19e72c" exitCode=0
Jul 24 01:01:14.101290 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.100954    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" event={"ID":"18bbdd4d-b45c-40e1-8842-265c126a6dce","Type":"ContainerDied","Data":"7153c017d59862ad32ffeaeb82add09c16db6ef44844386ad31072f8bf19e72c"}
Jul 24 01:01:14.101290 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.101036    2574 scope.go:117] "RemoveContainer" containerID="b3effc22cb35e488d4924d24f79b81ffd32c052c289781b5a1cc6c3f66598f41"
Jul 24 01:01:14.200189 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.200162    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 01:01:14.388757 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.388721    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-data\") pod \"18bbdd4d-b45c-40e1-8842-265c126a6dce\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") "
Jul 24 01:01:14.388941 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.388781    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-podinfo\") pod \"18bbdd4d-b45c-40e1-8842-265c126a6dce\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") "
Jul 24 01:01:14.388941 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.388829    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-socket\") pod \"18bbdd4d-b45c-40e1-8842-265c126a6dce\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") "
Jul 24 01:01:14.388941 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.388854    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-token\") pod \"18bbdd4d-b45c-40e1-8842-265c126a6dce\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") "
Jul 24 01:01:14.388941 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.388875    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-credential-socket\") pod \"18bbdd4d-b45c-40e1-8842-265c126a6dce\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") "
Jul 24 01:01:14.388941 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.388918    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-envoy\") pod \"18bbdd4d-b45c-40e1-8842-265c126a6dce\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") "
Jul 24 01:01:14.389194 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.388947    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18bbdd4d-b45c-40e1-8842-265c126a6dce-istiod-ca-cert\") pod \"18bbdd4d-b45c-40e1-8842-265c126a6dce\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") "
Jul 24 01:01:14.389194 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.388973    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-certs\") pod \"18bbdd4d-b45c-40e1-8842-265c126a6dce\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") "
Jul 24 01:01:14.389194 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.389004    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54vkf\" (UniqueName: \"kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-kube-api-access-54vkf\") pod \"18bbdd4d-b45c-40e1-8842-265c126a6dce\" (UID: \"18bbdd4d-b45c-40e1-8842-265c126a6dce\") "
Jul 24 01:01:14.389194 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.389097    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18bbdd4d-b45c-40e1-8842-265c126a6dce" (UID: "18bbdd4d-b45c-40e1-8842-265c126a6dce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:01:14.389451 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.389295    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:01:14.389451 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.389339    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18bbdd4d-b45c-40e1-8842-265c126a6dce" (UID: "18bbdd4d-b45c-40e1-8842-265c126a6dce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:01:14.389562 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.389490    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18bbdd4d-b45c-40e1-8842-265c126a6dce" (UID: "18bbdd4d-b45c-40e1-8842-265c126a6dce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:01:14.389776 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.389757    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-data" (OuterVolumeSpecName: "istio-data") pod "18bbdd4d-b45c-40e1-8842-265c126a6dce" (UID: "18bbdd4d-b45c-40e1-8842-265c126a6dce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:01:14.389776 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.389754    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bbdd4d-b45c-40e1-8842-265c126a6dce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18bbdd4d-b45c-40e1-8842-265c126a6dce" (UID: "18bbdd4d-b45c-40e1-8842-265c126a6dce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:01:14.391170 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.391141    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-token" (OuterVolumeSpecName: "istio-token") pod "18bbdd4d-b45c-40e1-8842-265c126a6dce" (UID: "18bbdd4d-b45c-40e1-8842-265c126a6dce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:01:14.391268 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.391226    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18bbdd4d-b45c-40e1-8842-265c126a6dce" (UID: "18bbdd4d-b45c-40e1-8842-265c126a6dce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:01:14.391549 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.391525    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18bbdd4d-b45c-40e1-8842-265c126a6dce" (UID: "18bbdd4d-b45c-40e1-8842-265c126a6dce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 01:01:14.391611 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.391526    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-kube-api-access-54vkf" (OuterVolumeSpecName: "kube-api-access-54vkf") pod "18bbdd4d-b45c-40e1-8842-265c126a6dce" (UID: "18bbdd4d-b45c-40e1-8842-265c126a6dce"). InnerVolumeSpecName "kube-api-access-54vkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:01:14.489768 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.489715    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-token\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:01:14.489768 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.489761    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-credential-socket\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:01:14.489768 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.489771    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-envoy\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:01:14.489768 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.489781    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18bbdd4d-b45c-40e1-8842-265c126a6dce-istiod-ca-cert\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:01:14.489768 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.489789    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-workload-certs\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:01:14.490083 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.489797    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54vkf\" (UniqueName: \"kubernetes.io/projected/18bbdd4d-b45c-40e1-8842-265c126a6dce-kube-api-access-54vkf\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:01:14.490083 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.489807    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-data\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:01:14.490083 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:14.489815    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18bbdd4d-b45c-40e1-8842-265c126a6dce-istio-podinfo\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:01:15.107334 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:15.107286    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"
Jul 24 01:01:15.107334 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:15.107304    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s" event={"ID":"18bbdd4d-b45c-40e1-8842-265c126a6dce","Type":"ContainerDied","Data":"31685c2191bde5daa495282ff84d9cb813f2aad636706a966aba75acda3aa455"}
Jul 24 01:01:15.107811 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:15.107387    2574 scope.go:117] "RemoveContainer" containerID="7153c017d59862ad32ffeaeb82add09c16db6ef44844386ad31072f8bf19e72c"
Jul 24 01:01:15.128137 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:15.128109    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"]
Jul 24 01:01:15.131850 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:15.131828    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-44952a-openshift-default-5d78f6f47b-gqt7s"]
Jul 24 01:01:15.753420 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:15.753384    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" path="/var/lib/kubelet/pods/18bbdd4d-b45c-40e1-8842-265c126a6dce/volumes"
Jul 24 01:01:19.802908 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:19.802874    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"]
Jul 24 01:01:19.803369 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:19.803088    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc" podUID="ea465a24-914a-4b7c-af9d-12e7865db575" containerName="payload-pre-processing" containerID="cri-o://b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b" gracePeriod=30
Jul 24 01:01:20.062464 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.062406    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 01:01:20.136030 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.135998    2574 generic.go:356] "Generic (PLEG): container finished" podID="ea465a24-914a-4b7c-af9d-12e7865db575" containerID="b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b" exitCode=0
Jul 24 01:01:20.136220 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.136044    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"
Jul 24 01:01:20.136220 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.136072    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc" event={"ID":"ea465a24-914a-4b7c-af9d-12e7865db575","Type":"ContainerDied","Data":"b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b"}
Jul 24 01:01:20.136220 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.136112    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc" event={"ID":"ea465a24-914a-4b7c-af9d-12e7865db575","Type":"ContainerDied","Data":"e295e1a501e9cfae0c6725c36a53ef53a49f49a35894200757b21a94530ec789"}
Jul 24 01:01:20.136220 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.136135    2574 scope.go:117] "RemoveContainer" containerID="b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b"
Jul 24 01:01:20.140745 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.140726    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwm5x\" (UniqueName: \"kubernetes.io/projected/ea465a24-914a-4b7c-af9d-12e7865db575-kube-api-access-mwm5x\") pod \"ea465a24-914a-4b7c-af9d-12e7865db575\" (UID: \"ea465a24-914a-4b7c-af9d-12e7865db575\") "
Jul 24 01:01:20.140853 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.140809    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea465a24-914a-4b7c-af9d-12e7865db575-plugins-config-volume\") pod \"ea465a24-914a-4b7c-af9d-12e7865db575\" (UID: \"ea465a24-914a-4b7c-af9d-12e7865db575\") "
Jul 24 01:01:20.141237 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.141210    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea465a24-914a-4b7c-af9d-12e7865db575-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea465a24-914a-4b7c-af9d-12e7865db575" (UID: "ea465a24-914a-4b7c-af9d-12e7865db575"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:01:20.142736 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.142707    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea465a24-914a-4b7c-af9d-12e7865db575-kube-api-access-mwm5x" (OuterVolumeSpecName: "kube-api-access-mwm5x") pod "ea465a24-914a-4b7c-af9d-12e7865db575" (UID: "ea465a24-914a-4b7c-af9d-12e7865db575"). InnerVolumeSpecName "kube-api-access-mwm5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:01:20.144012 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.143995    2574 scope.go:117] "RemoveContainer" containerID="b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b"
Jul 24 01:01:20.144266 ip-10-0-139-9 kubenswrapper[2574]: E0724 01:01:20.144247    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b\": container with ID starting with b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b not found: ID does not exist" containerID="b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b"
Jul 24 01:01:20.144323 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.144272    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b"} err="failed to get container status \"b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b\": rpc error: code = NotFound desc = could not find container \"b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b\": container with ID starting with b0b9a8f109c0f5b2b6ebdef2c729ae7914a3ed681fdbe0111f09ab09ff4ecb6b not found: ID does not exist"
Jul 24 01:01:20.241859 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.241821    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwm5x\" (UniqueName: \"kubernetes.io/projected/ea465a24-914a-4b7c-af9d-12e7865db575-kube-api-access-mwm5x\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:01:20.241859 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.241853    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea465a24-914a-4b7c-af9d-12e7865db575-plugins-config-volume\") on node \"ip-10-0-139-9.ec2.internal\" DevicePath \"\""
Jul 24 01:01:20.456511 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.456477    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"]
Jul 24 01:01:20.459105 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:20.459078    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-91a75b-55ffcbd576-vgpnc"]
Jul 24 01:01:21.751610 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:21.751572    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea465a24-914a-4b7c-af9d-12e7865db575" path="/var/lib/kubelet/pods/ea465a24-914a-4b7c-af9d-12e7865db575/volumes"
Jul 24 01:01:31.580872 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:31.580782    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bcf8b5879-srpzq_9e0f2154-96bd-4912-baca-59499712d479/authorino/0.log"
Jul 24 01:01:37.981813 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:37.981777    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cdb55f47c-ssxwz_2583d812-d620-4302-a7ae-d9aa54fc30fc/copy-manifests/0.log"
Jul 24 01:01:37.988915 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:37.988894    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cdb55f47c-ssxwz_2583d812-d620-4302-a7ae-d9aa54fc30fc/manager/0.log"
Jul 24 01:01:40.197457 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:40.197424    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bcf8b5879-srpzq_9e0f2154-96bd-4912-baca-59499712d479/authorino/0.log"
Jul 24 01:01:41.769554 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:41.769523    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dc97f664-qqqdd_49d70df8-bf64-4fb9-88e0-b9955f896137/kube-auth-proxy/0.log"
Jul 24 01:01:41.891039 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:41.891008    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mfq7p_a88fbbbb-bb27-4049-99b0-31b75cb659ed/istio-proxy/1.log"
Jul 24 01:01:42.011041 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:42.011011    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zs5fj_eb80b996-52a1-401c-beb3-9fe9b8ae3d54/payload-pre-processing/0.log"
Jul 24 01:01:43.061976 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:43.061938    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4df2v_22463cb3-f2eb-4781-9715-8ff4595faf9f/main/0.log"
Jul 24 01:01:43.179451 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:43.179412    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-7jc6p_c1ae7dfd-449c-4728-a134-d1271b09cdae/main/0.log"
Jul 24 01:01:48.507895 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.507863    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7v8jb/must-gather-4q6wm"]
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508124    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" containerName="main"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508134    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" containerName="main"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508143    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508149    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508156    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941b00f1-fd05-4185-b45a-88848cae5210" containerName="payload-pre-processing"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508162    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="941b00f1-fd05-4185-b45a-88848cae5210" containerName="payload-pre-processing"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508170    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" containerName="istio-proxy"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508176    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" containerName="istio-proxy"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508183    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84641ab4-14d6-4523-8aa8-5b865207d6c5" containerName="maas-api"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508191    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="84641ab4-14d6-4523-8aa8-5b865207d6c5" containerName="maas-api"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508197    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" containerName="istio-proxy"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508203    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" containerName="istio-proxy"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508210    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3" containerName="payload-processing"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508215    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3" containerName="payload-processing"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508227    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1793562-f014-47a1-9161-c3ae18439fde" containerName="maas-api"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508232    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1793562-f014-47a1-9161-c3ae18439fde" containerName="maas-api"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508240    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d58674b1-ba18-40f1-a58e-0cc79966a06c" containerName="payload-processing"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508245    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d58674b1-ba18-40f1-a58e-0cc79966a06c" containerName="payload-processing"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508254    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea465a24-914a-4b7c-af9d-12e7865db575" containerName="payload-pre-processing"
Jul 24 01:01:48.508263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508259    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea465a24-914a-4b7c-af9d-12e7865db575" containerName="payload-pre-processing"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508301    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="941b00f1-fd05-4185-b45a-88848cae5210" containerName="payload-pre-processing"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508309    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d58674b1-ba18-40f1-a58e-0cc79966a06c" containerName="payload-processing"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508329    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508334    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe8c1d66-20fb-4ff9-b49e-1cd5dfa86676" containerName="main"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508340    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea465a24-914a-4b7c-af9d-12e7865db575" containerName="payload-pre-processing"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508346    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" containerName="istio-proxy"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508351    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="84641ab4-14d6-4523-8aa8-5b865207d6c5" containerName="maas-api"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508356    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4dbbb6c-5bef-44fb-b86c-1d14c425c3c3" containerName="payload-processing"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508361    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1793562-f014-47a1-9161-c3ae18439fde" containerName="maas-api"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508431    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508437    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508477    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="18bbdd4d-b45c-40e1-8842-265c126a6dce" containerName="istio-proxy"
Jul 24 01:01:48.508951 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.508483    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a6af879-5324-4675-970d-83da0dd16b6e" containerName="istio-proxy"
Jul 24 01:01:48.512862 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.512845    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v8jb/must-gather-4q6wm"
Jul 24 01:01:48.515330 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.515282    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v8jb\"/\"openshift-service-ca.crt\""
Jul 24 01:01:48.515439 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.515360    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v8jb\"/\"kube-root-ca.crt\""
Jul 24 01:01:48.515439 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.515407    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7v8jb\"/\"default-dockercfg-hlg9z\""
Jul 24 01:01:48.520128 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.520099    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v8jb/must-gather-4q6wm"]
Jul 24 01:01:48.576122 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.576087    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qb7b\" (UniqueName: \"kubernetes.io/projected/1d561d2f-fddf-404e-bc6b-fd3c06a90bff-kube-api-access-5qb7b\") pod \"must-gather-4q6wm\" (UID: \"1d561d2f-fddf-404e-bc6b-fd3c06a90bff\") " pod="openshift-must-gather-7v8jb/must-gather-4q6wm"
Jul 24 01:01:48.576293 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.576184    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d561d2f-fddf-404e-bc6b-fd3c06a90bff-must-gather-output\") pod \"must-gather-4q6wm\" (UID: \"1d561d2f-fddf-404e-bc6b-fd3c06a90bff\") " pod="openshift-must-gather-7v8jb/must-gather-4q6wm"
Jul 24 01:01:48.676671 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.676635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qb7b\" (UniqueName: \"kubernetes.io/projected/1d561d2f-fddf-404e-bc6b-fd3c06a90bff-kube-api-access-5qb7b\") pod \"must-gather-4q6wm\" (UID: \"1d561d2f-fddf-404e-bc6b-fd3c06a90bff\") " pod="openshift-must-gather-7v8jb/must-gather-4q6wm"
Jul 24 01:01:48.676827 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.676697    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d561d2f-fddf-404e-bc6b-fd3c06a90bff-must-gather-output\") pod \"must-gather-4q6wm\" (UID: \"1d561d2f-fddf-404e-bc6b-fd3c06a90bff\") " pod="openshift-must-gather-7v8jb/must-gather-4q6wm"
Jul 24 01:01:48.676969 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.676955    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d561d2f-fddf-404e-bc6b-fd3c06a90bff-must-gather-output\") pod \"must-gather-4q6wm\" (UID: \"1d561d2f-fddf-404e-bc6b-fd3c06a90bff\") " pod="openshift-must-gather-7v8jb/must-gather-4q6wm"
Jul 24 01:01:48.684246 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.684220    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qb7b\" (UniqueName: \"kubernetes.io/projected/1d561d2f-fddf-404e-bc6b-fd3c06a90bff-kube-api-access-5qb7b\") pod \"must-gather-4q6wm\" (UID: \"1d561d2f-fddf-404e-bc6b-fd3c06a90bff\") " pod="openshift-must-gather-7v8jb/must-gather-4q6wm"
Jul 24 01:01:48.823418 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.823381    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v8jb/must-gather-4q6wm"
Jul 24 01:01:48.946816 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:48.946787    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v8jb/must-gather-4q6wm"]
Jul 24 01:01:48.949246 ip-10-0-139-9 kubenswrapper[2574]: W0724 01:01:48.949219    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d561d2f_fddf_404e_bc6b_fd3c06a90bff.slice/crio-ee304b09a64fd57bf2791c466119df9e3596c1c16afe03ca514b1005103f0ef3 WatchSource:0}: Error finding container ee304b09a64fd57bf2791c466119df9e3596c1c16afe03ca514b1005103f0ef3: Status 404 returned error can't find the container with id ee304b09a64fd57bf2791c466119df9e3596c1c16afe03ca514b1005103f0ef3
Jul 24 01:01:49.250257 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:49.250174    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v8jb/must-gather-4q6wm" event={"ID":"1d561d2f-fddf-404e-bc6b-fd3c06a90bff","Type":"ContainerStarted","Data":"ee304b09a64fd57bf2791c466119df9e3596c1c16afe03ca514b1005103f0ef3"}
Jul 24 01:01:53.273033 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:53.272182    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v8jb/must-gather-4q6wm" event={"ID":"1d561d2f-fddf-404e-bc6b-fd3c06a90bff","Type":"ContainerStarted","Data":"1d442b2f758c42d7c6d5e27b395be71632808238ac441ae26ac0977efc09c22b"}
Jul 24 01:01:53.273033 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:53.272234    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v8jb/must-gather-4q6wm" event={"ID":"1d561d2f-fddf-404e-bc6b-fd3c06a90bff","Type":"ContainerStarted","Data":"6e68dad388121c8de976b8fd76c6e567fd6e215c283c17f9e3e36e9bb9ec6d66"}
Jul 24 01:01:53.302979 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:53.302914    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7v8jb/must-gather-4q6wm" podStartSLOduration=1.773458358 podStartE2EDuration="5.302896091s" totalImagesPullingTime="3.529437733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:01:53.301704669 +0000 UTC m=+3076.150023412" watchObservedRunningTime="2026-07-24 01:01:53.302896091 +0000 UTC m=+3076.151214842"
Jul 24 01:01:54.202464 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:54.202428    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j26n7_a256a9ba-91cf-47ba-8e3b-a987081eac63/global-pull-secret-syncer/0.log"
Jul 24 01:01:54.368501 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:54.368462    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p7v47_6220ffd0-9ea8-41c3-b1c0-fa00c590bfb1/konnectivity-agent/0.log"
Jul 24 01:01:54.442956 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:54.442926    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-9.ec2.internal_adf7019e1593e3ce75a451d35149a13f/haproxy/0.log"
Jul 24 01:01:59.550358 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:01:59.549916    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bcf8b5879-srpzq_9e0f2154-96bd-4912-baca-59499712d479/authorino/0.log"
Jul 24 01:02:01.946715 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:01.946676    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8vlz6_a4c8622d-3eb6-4d2c-b71e-0866019ceebd/node-exporter/0.log"
Jul 24 01:02:01.971352 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:01.971307    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8vlz6_a4c8622d-3eb6-4d2c-b71e-0866019ceebd/kube-rbac-proxy/0.log"
Jul 24 01:02:01.996072 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:01.996047    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8vlz6_a4c8622d-3eb6-4d2c-b71e-0866019ceebd/init-textfile/0.log"
Jul 24 01:02:04.455872 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.455832    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"]
Jul 24 01:02:04.461100 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.461073    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.465269 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.465234    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"]
Jul 24 01:02:04.632998 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.632957    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-proc\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.633196 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.633018    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr62n\" (UniqueName: \"kubernetes.io/projected/6e6c931e-9713-4502-b00d-5e9a76875d3f-kube-api-access-fr62n\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.633196 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.633086    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-sys\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.633196 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.633133    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-lib-modules\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.633196 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.633157    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-podres\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.734090 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.734006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-proc\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.734090 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.734075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr62n\" (UniqueName: \"kubernetes.io/projected/6e6c931e-9713-4502-b00d-5e9a76875d3f-kube-api-access-fr62n\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.734290 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.734109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-sys\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.734290 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.734133    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-lib-modules\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.734290 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.734141    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-proc\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.734290 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.734154    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-podres\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.734290 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.734250    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-sys\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.734545 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.734380    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-podres\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.734545 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.734403    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e6c931e-9713-4502-b00d-5e9a76875d3f-lib-modules\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.743215 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.743188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr62n\" (UniqueName: \"kubernetes.io/projected/6e6c931e-9713-4502-b00d-5e9a76875d3f-kube-api-access-fr62n\") pod \"perf-node-gather-daemonset-xxpj6\" (UID: \"6e6c931e-9713-4502-b00d-5e9a76875d3f\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.796405 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.796160    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:04.971251 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:04.971146    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"]
Jul 24 01:02:04.977627 ip-10-0-139-9 kubenswrapper[2574]: W0724 01:02:04.977596    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6e6c931e_9713_4502_b00d_5e9a76875d3f.slice/crio-133a73ed5f2ce3cb68234bf7cd4bb27bb2ad5c29328b03dddcf57ddca301d9b0 WatchSource:0}: Error finding container 133a73ed5f2ce3cb68234bf7cd4bb27bb2ad5c29328b03dddcf57ddca301d9b0: Status 404 returned error can't find the container with id 133a73ed5f2ce3cb68234bf7cd4bb27bb2ad5c29328b03dddcf57ddca301d9b0
Jul 24 01:02:05.353504 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:05.353460    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6" event={"ID":"6e6c931e-9713-4502-b00d-5e9a76875d3f","Type":"ContainerStarted","Data":"a51de62be4ce9d9453ea2a1e5dd408f5b6fea290fe6e5cc4a6df347d1ff4661b"}
Jul 24 01:02:05.353696 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:05.353530    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6" event={"ID":"6e6c931e-9713-4502-b00d-5e9a76875d3f","Type":"ContainerStarted","Data":"133a73ed5f2ce3cb68234bf7cd4bb27bb2ad5c29328b03dddcf57ddca301d9b0"}
Jul 24 01:02:05.353696 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:05.353574    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:05.370939 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:05.370877    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6" podStartSLOduration=1.370857516 podStartE2EDuration="1.370857516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:02:05.36864887 +0000 UTC m=+3088.216967612" watchObservedRunningTime="2026-07-24 01:02:05.370857516 +0000 UTC m=+3088.219176258"
Jul 24 01:02:06.831910 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:06.831876    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vqjhl_909f0b40-b312-413e-864f-5acc0121e6d6/dns/0.log"
Jul 24 01:02:06.858479 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:06.858439    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vqjhl_909f0b40-b312-413e-864f-5acc0121e6d6/kube-rbac-proxy/0.log"
Jul 24 01:02:06.887603 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:06.887577    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ksmwx_6364a346-b501-4735-a5c3-97ee5b0120af/dns-node-resolver/0.log"
Jul 24 01:02:07.558507 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:07.558475    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-66ffc854c8-5t479_134e7585-1ffa-4f79-bc7d-bc599bc98893/registry/0.log"
Jul 24 01:02:07.606048 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:07.606018    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kdddd_f5b7f667-f9ea-4deb-9cb8-362fe65e017e/node-ca/0.log"
Jul 24 01:02:08.759837 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:08.759802    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dc97f664-qqqdd_49d70df8-bf64-4fb9-88e0-b9955f896137/kube-auth-proxy/0.log"
Jul 24 01:02:08.790351 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:08.790305    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mfq7p_a88fbbbb-bb27-4049-99b0-31b75cb659ed/istio-proxy/0.log"
Jul 24 01:02:08.825487 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:08.825456    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mfq7p_a88fbbbb-bb27-4049-99b0-31b75cb659ed/istio-proxy/1.log"
Jul 24 01:02:08.865144 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:08.865113    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zs5fj_eb80b996-52a1-401c-beb3-9fe9b8ae3d54/payload-pre-processing/0.log"
Jul 24 01:02:09.742076 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:09.742049    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jskz7_c04cdbd1-5fa0-4813-95eb-fc69a980dd2e/serve-healthcheck-canary/0.log"
Jul 24 01:02:10.249019 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:10.248996    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-72lwd_418b27c1-b425-4aed-a5e1-1f59b7a817b2/kube-rbac-proxy/0.log"
Jul 24 01:02:10.273238 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:10.273211    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-72lwd_418b27c1-b425-4aed-a5e1-1f59b7a817b2/exporter/0.log"
Jul 24 01:02:10.299488 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:10.299457    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-72lwd_418b27c1-b425-4aed-a5e1-1f59b7a817b2/extractor/0.log"
Jul 24 01:02:11.370719 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:11.370690    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-xxpj6"
Jul 24 01:02:13.158358 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:13.158332    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cdb55f47c-ssxwz_2583d812-d620-4302-a7ae-d9aa54fc30fc/manager/0.log"
Jul 24 01:02:13.182000 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:13.181960    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cdb55f47c-ssxwz_2583d812-d620-4302-a7ae-d9aa54fc30fc/copy-manifests/0.log"
Jul 24 01:02:14.638366 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:14.638334    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5bbf775ccb-rh6mb_ae5e41e2-699c-42ce-91ef-453faac576fc/manager/0.log"
Jul 24 01:02:14.967263 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:14.967236    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dc97f664-qqqdd_49d70df8-bf64-4fb9-88e0-b9955f896137/kube-auth-proxy/0.log"
Jul 24 01:02:14.986244 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:14.986214    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mfq7p_a88fbbbb-bb27-4049-99b0-31b75cb659ed/istio-proxy/0.log"
Jul 24 01:02:15.015375 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:15.015344    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mfq7p_a88fbbbb-bb27-4049-99b0-31b75cb659ed/istio-proxy/1.log"
Jul 24 01:02:15.048281 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:15.048251    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zs5fj_eb80b996-52a1-401c-beb3-9fe9b8ae3d54/payload-pre-processing/0.log"
Jul 24 01:02:16.723480 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:16.723413    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dc97f664-qqqdd_49d70df8-bf64-4fb9-88e0-b9955f896137/kube-auth-proxy/0.log"
Jul 24 01:02:16.758847 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:16.758822    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mfq7p_a88fbbbb-bb27-4049-99b0-31b75cb659ed/istio-proxy/0.log"
Jul 24 01:02:16.789351 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:16.789300    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mfq7p_a88fbbbb-bb27-4049-99b0-31b75cb659ed/istio-proxy/1.log"
Jul 24 01:02:16.820534 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:16.820507    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-zs5fj_eb80b996-52a1-401c-beb3-9fe9b8ae3d54/payload-pre-processing/0.log"
Jul 24 01:02:21.560284 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:21.560201    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vh7wc_acd4b7a0-11ee-40fb-ba36-9fb664430562/kube-multus-additional-cni-plugins/0.log"
Jul 24 01:02:21.588013 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:21.587981    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vh7wc_acd4b7a0-11ee-40fb-ba36-9fb664430562/egress-router-binary-copy/0.log"
Jul 24 01:02:21.614035 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:21.614009    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vh7wc_acd4b7a0-11ee-40fb-ba36-9fb664430562/cni-plugins/0.log"
Jul 24 01:02:21.638984 ip-10-0-139-9 kubenswrapper[2574]: I0724 01:02:21.638944    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vh7wc_acd4b7a0-11ee-40fb-ba36-9fb664430562/bond-cni-plugin/0.log"