Jul 24 00:13:39.293161 ip-10-0-135-176 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.747227    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754093    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754112    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754117    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754122    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754128    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754134    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754139    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:13:39.810524 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754143    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754147    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754151    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754155    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754159    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754163    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754167    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754171    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754175    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754178    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754182    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754186    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754190    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754194    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754198    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754202    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754206    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754220    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754224    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754229    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:13:40.042882 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754232    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:13:39.835991 ip-10-0-135-176 systemd[1]: Started Kubernetes Kubelet.
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754236    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754240    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754243    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754247    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754251    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754256    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754260    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754264    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754268    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754272    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754276    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754280    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754283    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754287    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754291    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754295    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754299    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754303    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754307    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754312    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:13:40.057119 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754316    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754320    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754323    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754327    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754332    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754336    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754340    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754344    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754347    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754351    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754355    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754359    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754363    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754366    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754370    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754374    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754378    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754384    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754388    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:13:40.058131 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754392    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754397    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754400    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754405    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754409    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754412    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754416    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754421    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754430    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754435    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754440    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754444    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754449    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754454    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754458    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754462    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754466    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754470    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754474    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:13:40.059136 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754477    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754481    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754486    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754489    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754491    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754495    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754499    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754502    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754506    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754510    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754513    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754517    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.754521    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756476    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756484    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756488    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756492    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756495    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756498    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:13:40.059743 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756501    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756504    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756507    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756509    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756512    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756514    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756517    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756520    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756523    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756526    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756528    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756531    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756534    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756538    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756542    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756545    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756548    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756550    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756553    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:13:40.060347 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756556    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756558    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756561    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756564    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756566    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756569    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756571    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756574    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756576    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756579    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756582    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756584    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756587    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756589    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756592    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756595    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756598    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756601    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756603    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756606    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:13:40.061169 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756608    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756611    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756614    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756617    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756619    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756623    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756625    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756628    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756631    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756633    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756636    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756638    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756643    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756646    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756649    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756651    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756654    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756657    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756659    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756662    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:13:40.062073 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756664    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756667    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756669    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756672    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756674    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756677    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756679    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756681    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756684    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756687    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756689    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756692    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756694    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756697    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756700    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756703    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756705    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756708    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756710    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756712    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:13:40.062742 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756716    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756718    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756722    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756726    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756730    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756735    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756738    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756741    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756744    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756747    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756751    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756754    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756758    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.756760    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757163    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757168    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757171    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757173    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757177    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:13:40.063453 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757179    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757183    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757185    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757188    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757190    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757193    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757196    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757199    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757202    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757204    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757207    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757210    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757212    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757216    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757219    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757222    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757224    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757229    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757231    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757235    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:13:40.064057 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757239    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757241    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757244    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757247    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757250    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757253    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757256    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757259    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757261    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757265    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757268    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757271    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757274    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757276    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757279    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757281    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757284    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757287    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757289    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757292    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:13:40.064738 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757295    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757298    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757300    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757303    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757306    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757308    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757311    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757313    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757316    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757320    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757325    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757328    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757331    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757334    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757336    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757339    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757342    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757345    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757348    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:13:40.065344 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757351    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757353    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757356    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757359    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757361    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757364    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757366    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757369    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757371    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757374    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757377    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757381    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757383    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757386    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757390    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757392    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757395    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757397    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757400    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757402    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:13:40.066182 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757405    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757407    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757411    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757414    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757417    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757420    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757422    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757425    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757427    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757430    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757432    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757435    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757437    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757440    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.757442    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757517    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757533    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757542    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757549    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757555    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757560    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 00:13:40.066846 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757564    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757568    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757573    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757576    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757579    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757582    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757590    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757593    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757596    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757599    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757602    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757605    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757608    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757612    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757616    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757619    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757622    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757626    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757629    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757632    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757635    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757639    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757642    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757644    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 00:13:40.067479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757648    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757651    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757655    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757658    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757661    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757664    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757667    2560 flags.go:64] FLAG: --enable-server="true"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757670    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757677    2560 flags.go:64] FLAG: --event-burst="100"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757680    2560 flags.go:64] FLAG: --event-qps="50"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757686    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757689    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757692    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757696    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757699    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757704    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757707    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757710    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757713    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757716    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757719    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757723    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757726    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757728    2560 flags.go:64] FLAG: --feature-gates=""
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757732    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 00:13:40.068134 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757735    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757738    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757741    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757744    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757747    2560 flags.go:64] FLAG: --help="false"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757750    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757759    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757764    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757767    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757770    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757774    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757777    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757779    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757782    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757785    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757788    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757791    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757795    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757798    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757801    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757804    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757807    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757810    2560 flags.go:64] FLAG: --lock-file=""
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757813    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 00:13:40.069034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757816    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757819    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757824    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757839    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757845    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757848    2560 flags.go:64] FLAG: --logging-format="text"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757851    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757855    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757858    2560 flags.go:64] FLAG: --manifest-url=""
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757860    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757865    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757868    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757872    2560 flags.go:64] FLAG: --max-pods="110"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757875    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757877    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757880    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757883    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757886    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757889    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757892    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757899    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757902    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757905    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 00:13:40.069732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757908    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757911    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757918    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757921    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757924    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757927    2560 flags.go:64] FLAG: --port="10250"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757930    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757933    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-069b9c4ebe55cc8d5"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757936    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757939    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757942    2560 flags.go:64] FLAG: --register-node="true"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757945    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757949    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757952    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757956    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757959    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757963    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757966    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757969    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757971    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757974    2560 flags.go:64] FLAG: --runonce="false"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757977    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757980    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757983    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757986    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 00:13:40.070666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757989    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757992    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757995    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.757998    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758001    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758004    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758006    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758009    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758012    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758016    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758019    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758024    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758027    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758030    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758037    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758039    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758043    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758046    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758049    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758052    2560 flags.go:64] FLAG: --v="2"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758056    2560 flags.go:64] FLAG: --version="false"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758062    2560 flags.go:64] FLAG: --vmodule=""
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758065    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.758069    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758193    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:13:40.071729 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758197    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758202    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758206    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758208    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758212    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758214    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758217    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758220    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758222    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758225    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758228    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758231    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758233    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758236    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758238    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758241    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758245    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758247    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758250    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:13:40.072718 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758252    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758255    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758258    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758260    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758263    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758265    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758269    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758271    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758274    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758279    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758281    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758284    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758286    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758289    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758292    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758294    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758297    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758300    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758302    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758305    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:13:40.301491 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758308    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758310    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758313    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758315    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758318    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758320    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758323    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758326    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758328    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758332    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758334    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758337    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758341    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758344    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758346    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758349    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758352    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758354    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758357    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:13:40.302519 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758360    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758363    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758366    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758369    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758372    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758374    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758377    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758380    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758382    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758385    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758387    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758390    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758392    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758395    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758398    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758401    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758404    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758408    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758411    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758413    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:13:40.303059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758416    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758419    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758423    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758426    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758428    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758431    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758433    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758436    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758438    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758441    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758444    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758446    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758449    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758452    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758456    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758459    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758461    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758464    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758466    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:13:40.303625 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.758469    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.759144    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.766350    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.766367    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.766387    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.766393    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766449    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766454    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766457    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766461    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766465    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766469    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766473    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766476    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:13:40.304148 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766479    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766482    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766485    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766488    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766490    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766493    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766496    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766499    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766501    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766505    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766507    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766510    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766512    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766515    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766518    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766520    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766523    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766530    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766533    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:13:40.304564 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766536    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766539    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766542    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766546    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766549    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766552    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766555    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766558    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766561    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766565    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766568    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766571    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766574    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766577    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766580    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766583    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766585    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766588    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766591    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766594    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:13:40.305086 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766597    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766600    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766602    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766605    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766608    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766611    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766613    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766616    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766618    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766621    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766627    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766630    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766633    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766635    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766638    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766641    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766644    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766646    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766649    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766651    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:13:40.305632 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766654    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766656    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766659    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766662    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766664    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766667    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766669    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766672    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766675    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766677    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766680    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766683    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766685    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766688    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766690    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766692    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766695    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766698    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766700    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766703    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:13:40.306197 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766706    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766708    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766713    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766716    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766718    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766721    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766724    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766726    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766729    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766732    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766734    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766737    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.766742    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766889    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766895    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 00:13:40.306756 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766898    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766901    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766904    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766908    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766912    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766915    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766918    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766921    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766923    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766926    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766929    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766931    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766934    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766937    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766939    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766942    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766945    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766947    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766950    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766954    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 00:13:40.307186 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766958    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766960    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766963    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766965    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766968    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766971    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766974    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766976    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766979    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766981    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766984    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766987    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766991    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766994    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.766997    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767000    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767003    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767005    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767008    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 00:13:40.307748 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767011    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767013    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767016    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767018    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767021    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767024    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767026    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767028    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767031    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767034    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767036    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767039    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767043    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767047    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767050    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767053    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767055    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767058    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767061    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 00:13:40.308270 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767063    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767066    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767069    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767072    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767074    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767076    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767079    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767082    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767084    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767087    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767090    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767092    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767095    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767097    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767100    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767102    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767105    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767107    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767110    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767112    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 00:13:40.308792 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767115    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767117    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767120    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767122    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767125    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767128    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767132    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767135    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767138    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767140    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767144    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767148    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767151    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767154    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767157    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767159    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767162    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767164    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 00:13:40.309341 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:39.767167    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.767171    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.767307    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.771120    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.772744    2560 server.go:1019] "Starting client certificate rotation"
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.772862    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.772908    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.776772    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.781641    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.797790    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.800269    2560 log.go:25] "Validated CRI v1 image API"
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.801094    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 00:13:40.309908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.805667    2560 fs.go:135] Filesystem UUIDs: map[14d19d8f-c4eb-4483-a259-9d2e5aebd25f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d41ea0b5-944c-4f67-a760-320c27606ed9:/dev/nvme0n1p4]
Jul 24 00:13:40.310250 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.805698    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 00:13:40.310250 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.806849    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.812627    2560 manager.go:216] Machine: {Timestamp:2026-07-24 00:13:39.811153384 +0000 UTC m=+0.378969525 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099016 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28e590872ea2b53ab106b0e7385ae0 SystemUUID:ec28e590-872e-a2b5-3ab1-06b0e7385ae0 BootID:2ddadf44-c9e4-4737-82f4-dc1bfa469146 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:70:9a:59:76:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:70:9a:59:76:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:7c:fc:15:de:37 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.812734    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.812818    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.816504    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.816530    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-176.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.817342    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.817351    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.817359    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.817374    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.818581    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.821447    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.821566    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.821594    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.821603    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.821622    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.821631    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.821640    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 00:13:40.310422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.822043    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.822058    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.825077    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.827635    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.827663    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829699    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829713    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829720    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829725    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829735    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829741    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829751    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829765    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829773    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829778    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829788    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.829794    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.831005    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.831013    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.833063    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.833063    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.833209    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-176.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.835055    2560 server.go:1268] "Started kubelet"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.835267    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.835309    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.836040    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.838176    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.841658    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.842411    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.848573    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5bxch"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.847933    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-176.ec2.internal.18c511749ae7aa9c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-176.ec2.internal,UID:ip-10-0-135-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-176.ec2.internal,},FirstTimestamp:2026-07-24 00:13:39.83502198 +0000 UTC m=+0.402838122,LastTimestamp:2026-07-24 00:13:39.83502198 +0000 UTC m=+0.402838122,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-176.ec2.internal,}"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.849378    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.849385    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.849501    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850148    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850147    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850191    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850307    2560 factory.go:55] Registering systemd factory
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850323    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850334    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850344    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 24 00:13:40.311071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850598    2560 factory.go:153] Registering CRI-O factory
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850612    2560 factory.go:223] Registration of the crio container factory successfully
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850664    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850688    2560 factory.go:103] Registering Raw factory
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.850704    2560 manager.go:1196] Started watching for new ooms in manager
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.851175    2560 manager.go:349] Starting recovery of all containers
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.857102    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5bxch"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.862996    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.863093    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.864709    2560 manager.go:354] Recovery completed
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.868777    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.872526    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.872553    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.872564    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.873153    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.873165    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.873165    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.873183    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.874314    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.874366    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.874402    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.874643    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-176.ec2.internal.18c511749d2422d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-176.ec2.internal,UID:ip-10-0-135-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-176.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-176.ec2.internal,},FirstTimestamp:2026-07-24 00:13:39.87253935 +0000 UTC m=+0.440355490,LastTimestamp:2026-07-24 00:13:39.87253935 +0000 UTC m=+0.440355490,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-176.ec2.internal,}"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.875985    2560 policy_none.go:49] "None policy: Start"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.876002    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.876015    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.878055    2560 policy_none.go:47] "Start"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.911460    2560 manager.go:337] "Starting Device Plugin manager"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.911496    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.911507    2560 server.go:87] "Starting device plugin registration server"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.911798    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.911810    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.911926    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.912028    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.912040    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.912580    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.912631    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.959567    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.960806    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.960867    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.960898    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 00:13:40.491077 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:39.960961    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:39.963557    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.012795    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.014037    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.014086    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.014101    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.014132    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.025189    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.025218    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-176.ec2.internal\": node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.048236    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.061327    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal"]
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.061396    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.062912    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.062941    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.062953    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.065151    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.065317    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.065345    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.066091    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.066102    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.066119    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.066129    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.066129    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.066229    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.068186    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.068214    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.069166    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.069196    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.069208    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeHasSufficientPID"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.085389    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-176.ec2.internal\" not found" node="ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.088928    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-176.ec2.internal\" not found" node="ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.149237    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.151273    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18fde8ac8173b64a16e5b7537a21972f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.ec2.internal\" (UID: \"18fde8ac8173b64a16e5b7537a21972f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.493825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.151302    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c416aa043ef3695f72f46e3bfb88066-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1c416aa043ef3695f72f46e3bfb88066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.151320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c416aa043ef3695f72f46e3bfb88066-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1c416aa043ef3695f72f46e3bfb88066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.249381    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.251596    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18fde8ac8173b64a16e5b7537a21972f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.ec2.internal\" (UID: \"18fde8ac8173b64a16e5b7537a21972f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.251624    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c416aa043ef3695f72f46e3bfb88066-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1c416aa043ef3695f72f46e3bfb88066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.251641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c416aa043ef3695f72f46e3bfb88066-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1c416aa043ef3695f72f46e3bfb88066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.251668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c416aa043ef3695f72f46e3bfb88066-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1c416aa043ef3695f72f46e3bfb88066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.251704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c416aa043ef3695f72f46e3bfb88066-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal\" (UID: \"1c416aa043ef3695f72f46e3bfb88066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.251703    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18fde8ac8173b64a16e5b7537a21972f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-176.ec2.internal\" (UID: \"18fde8ac8173b64a16e5b7537a21972f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.349989    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.388332    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.392474    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:40.495473 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.450058    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.724103 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.550938    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.724103 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.651441    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.724103 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.692530    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:13:40.985212 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.751678    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.985212 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.772455    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 00:13:40.985212 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.772632    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 00:13:40.985212 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.772632    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 00:13:40.985212 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.852157    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:40.985212 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.863997    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 00:08:40 +0000 UTC" deadline="2028-02-20 06:51:54.416112343 +0000 UTC"
Jul 24 00:13:40.985212 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:40.864026    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13830h38m13.552088925s"
Jul 24 00:13:40.985212 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:40.952556    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:41.184934 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:41.053269    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:41.184934 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:41.153955    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:41.417734 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:41.254612    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-176.ec2.internal\" not found"
Jul 24 00:13:41.417734 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.282174    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:13:41.417734 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.326134    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:13:41.417734 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.350569    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:41.417734 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.365245    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 00:13:41.417734 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.367305    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal"
Jul 24 00:13:41.417734 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.376186    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 00:13:41.417734 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.398191    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.825932    2560 apiserver.go:52] "Watching apiserver"
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.831171    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.832413    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-xctk4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792","openshift-image-registry/node-ca-kd9fr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal","openshift-multus/multus-additional-cni-plugins-kw8qh","openshift-multus/multus-zjgvk","openshift-multus/network-metrics-daemon-7kps9","openshift-network-diagnostics/network-check-target-r722z","kube-system/global-pull-secret-syncer-kgsn8","kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal","openshift-cluster-node-tuning-operator/tuned-fz6zv","openshift-dns/node-resolver-mttpm","openshift-network-operator/iptables-alerter-66t22","openshift-ovn-kubernetes/ovnkube-node-62qhx"]
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.836447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-66t22"
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.838742    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.838762    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.838765    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.839037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-42ml2\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.839156    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.841077    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.841231    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kd9fr"
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.841449    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.841453    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.841542    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c4vct\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.843704    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.844266    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.844440    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.844697    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.845002    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zlm2t\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.845828    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.846253    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.846280    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9fc75\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.848354    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.850113    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.850947    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.852001    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.852096    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.852123    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xd2hx\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.852168    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.852174    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.852220    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.853063    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.853117    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:41.934903 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:41.853199    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.853340    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fkmfl\""
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.855559    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:41.855609    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858104    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b87ed47a-6f1c-4f68-ae51-11144c46d320-konnectivity-ca\") pod \"konnectivity-agent-xctk4\" (UID: \"b87ed47a-6f1c-4f68-ae51-11144c46d320\") " pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858141    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-cnibin\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:41.858169    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-socket-dir-parent\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858229    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-hostroot\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858280    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-conf-dir\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858311    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-system-cni-dir\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858340    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-system-cni-dir\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-var-lib-cni-bin\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-run-multus-certs\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dprz\" (UniqueName: \"kubernetes.io/projected/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-kube-api-access-6dprz\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-cni-binary-copy\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.937190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-etc-kubernetes\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d50d211-9d46-4e14-ace4-554592a1ddc0-host\") pod \"node-ca-kd9fr\" (UID: \"6d50d211-9d46-4e14-ace4-554592a1ddc0\") " pod="openshift-image-registry/node-ca-kd9fr"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-os-release\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858591    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-var-lib-cni-multus\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858607    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-socket-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858623    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d50d211-9d46-4e14-ace4-554592a1ddc0-serviceca\") pod \"node-ca-kd9fr\" (UID: \"6d50d211-9d46-4e14-ace4-554592a1ddc0\") " pod="openshift-image-registry/node-ca-kd9fr"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858639    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dgt4\" (UniqueName: \"kubernetes.io/projected/6d50d211-9d46-4e14-ace4-554592a1ddc0-kube-api-access-9dgt4\") pod \"node-ca-kd9fr\" (UID: \"6d50d211-9d46-4e14-ace4-554592a1ddc0\") " pod="openshift-image-registry/node-ca-kd9fr"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b87ed47a-6f1c-4f68-ae51-11144c46d320-agent-certs\") pod \"konnectivity-agent-xctk4\" (UID: \"b87ed47a-6f1c-4f68-ae51-11144c46d320\") " pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858690    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a888e7d-1f98-4081-a014-cca49462de1f-host-slash\") pod \"iptables-alerter-66t22\" (UID: \"2a888e7d-1f98-4081-a014-cca49462de1f\") " pod="openshift-network-operator/iptables-alerter-66t22"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858744    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-registration-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858770    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-os-release\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-cni-dir\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-var-lib-kubelet\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7j6d\" (UniqueName: \"kubernetes.io/projected/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-kube-api-access-s7j6d\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858872    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-device-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:41.938688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858897    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lsq2\" (UniqueName: \"kubernetes.io/projected/7b009975-4390-46dd-9c1a-609bb499b2c3-kube-api-access-9lsq2\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.858973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-cnibin\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.859002    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-daemon-config\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.859026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a888e7d-1f98-4081-a014-cca49462de1f-iptables-alerter-script\") pod \"iptables-alerter-66t22\" (UID: \"2a888e7d-1f98-4081-a014-cca49462de1f\") " pod="openshift-network-operator/iptables-alerter-66t22"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.859051    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.859073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.859094    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-sys-fs\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.859117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.859143    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-run-k8s-cni-cncf-io\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.859169    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-run-netns\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.859197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2zs2\" (UniqueName: \"kubernetes.io/projected/2a888e7d-1f98-4081-a014-cca49462de1f-kube-api-access-x2zs2\") pod \"iptables-alerter-66t22\" (UID: \"2a888e7d-1f98-4081-a014-cca49462de1f\") " pod="openshift-network-operator/iptables-alerter-66t22"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.860650    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.862965    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.862982    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.863126    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.863246    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.863298    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.863343    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6r6tj\""
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.863361    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.863530    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.864003    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.865552    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mttpm"
Jul 24 00:13:41.939509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.866320    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 00:13:41.940604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.866352    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 00:13:41.940604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.866484    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j92ws\""
Jul 24 00:13:41.940604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.867950    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8l6np\""
Jul 24 00:13:41.940604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.868119    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 00:13:41.940604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.868402    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 00:13:41.940604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.885592    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lkmxr"
Jul 24 00:13:41.940604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.894172    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lkmxr"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.952285    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959441    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-sysctl-d\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-system-cni-dir\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-node-log\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959679    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-system-cni-dir\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-sysconfig\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dprz\" (UniqueName: \"kubernetes.io/projected/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-kube-api-access-6dprz\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-etc-kubernetes\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71f46a43-3c16-45f8-90c4-0f815e6850d0-kubelet-config\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-ovn-node-metrics-cert\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959968    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwcx7\" (UniqueName: \"kubernetes.io/projected/55bd3079-c0b1-4319-9e19-566a47f07873-kube-api-access-lwcx7\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.959986    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-etc-kubernetes\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960004    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-slash\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-sysctl-conf\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.140882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d50d211-9d46-4e14-ace4-554592a1ddc0-serviceca\") pod \"node-ca-kd9fr\" (UID: \"6d50d211-9d46-4e14-ace4-554592a1ddc0\") " pod="openshift-image-registry/node-ca-kd9fr"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dgt4\" (UniqueName: \"kubernetes.io/projected/6d50d211-9d46-4e14-ace4-554592a1ddc0-kube-api-access-9dgt4\") pod \"node-ca-kd9fr\" (UID: \"6d50d211-9d46-4e14-ace4-554592a1ddc0\") " pod="openshift-image-registry/node-ca-kd9fr"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b87ed47a-6f1c-4f68-ae51-11144c46d320-agent-certs\") pod \"konnectivity-agent-xctk4\" (UID: \"b87ed47a-6f1c-4f68-ae51-11144c46d320\") " pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960213    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a888e7d-1f98-4081-a014-cca49462de1f-host-slash\") pod \"iptables-alerter-66t22\" (UID: \"2a888e7d-1f98-4081-a014-cca49462de1f\") " pod="openshift-network-operator/iptables-alerter-66t22"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960248    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-registration-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-modprobe-d\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-run\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960301    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a888e7d-1f98-4081-a014-cca49462de1f-host-slash\") pod \"iptables-alerter-66t22\" (UID: \"2a888e7d-1f98-4081-a014-cca49462de1f\") " pod="openshift-network-operator/iptables-alerter-66t22"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960355    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-registration-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-os-release\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-var-lib-kubelet\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7j6d\" (UniqueName: \"kubernetes.io/projected/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-kube-api-access-s7j6d\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960500    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-os-release\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960511    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lsq2\" (UniqueName: \"kubernetes.io/projected/7b009975-4390-46dd-9c1a-609bb499b2c3-kube-api-access-9lsq2\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-var-lib-kubelet\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.142021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-log-socket\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-systemd\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960572    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a00cf2c5-5c20-410f-bae2-80170581b317-hosts-file\") pod \"node-resolver-mttpm\" (UID: \"a00cf2c5-5c20-410f-bae2-80170581b317\") " pod="openshift-dns/node-resolver-mttpm"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a00cf2c5-5c20-410f-bae2-80170581b317-tmp-dir\") pod \"node-resolver-mttpm\" (UID: \"a00cf2c5-5c20-410f-bae2-80170581b317\") " pod="openshift-dns/node-resolver-mttpm"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a888e7d-1f98-4081-a014-cca49462de1f-iptables-alerter-script\") pod \"iptables-alerter-66t22\" (UID: \"2a888e7d-1f98-4081-a014-cca49462de1f\") " pod="openshift-network-operator/iptables-alerter-66t22"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960706    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-sys-fs\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960717    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d50d211-9d46-4e14-ace4-554592a1ddc0-serviceca\") pod \"node-ca-kd9fr\" (UID: \"6d50d211-9d46-4e14-ace4-554592a1ddc0\") " pod="openshift-image-registry/node-ca-kd9fr"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-sys-fs\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960843    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-run-ovn\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960873    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960883    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-host\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.960984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5xhk\" (UniqueName: \"kubernetes.io/projected/a00cf2c5-5c20-410f-bae2-80170581b317-kube-api-access-b5xhk\") pod \"node-resolver-mttpm\" (UID: \"a00cf2c5-5c20-410f-bae2-80170581b317\") " pod="openshift-dns/node-resolver-mttpm"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2zs2\" (UniqueName: \"kubernetes.io/projected/2a888e7d-1f98-4081-a014-cca49462de1f-kube-api-access-x2zs2\") pod \"iptables-alerter-66t22\" (UID: \"2a888e7d-1f98-4081-a014-cca49462de1f\") " pod="openshift-network-operator/iptables-alerter-66t22"
Jul 24 00:13:42.143832 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961104    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2xb6\" (UniqueName: \"kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6\") pod \"network-check-target-r722z\" (UID: \"1bae97a1-6b3a-4ff1-847b-2df09e067715\") " pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9fkw\" (UniqueName: \"kubernetes.io/projected/b63b995d-5404-4953-acf6-31e068d8cd35-kube-api-access-v9fkw\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-cnibin\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961263    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-socket-dir-parent\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-hostroot\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a888e7d-1f98-4081-a014-cca49462de1f-iptables-alerter-script\") pod \"iptables-alerter-66t22\" (UID: \"2a888e7d-1f98-4081-a014-cca49462de1f\") " pod="openshift-network-operator/iptables-alerter-66t22"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-conf-dir\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-cnibin\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-kubernetes\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961367    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-hostroot\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-conf-dir\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-system-cni-dir\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-socket-dir-parent\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-var-lib-cni-bin\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961440    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-system-cni-dir\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-run-multus-certs\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.144517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961467    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-var-lib-cni-bin\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-run-netns\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961496    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-run-multus-certs\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961508    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t44gp\" (UniqueName: \"kubernetes.io/projected/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-kube-api-access-t44gp\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-sys\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961553    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55bd3079-c0b1-4319-9e19-566a47f07873-etc-tuned\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-cni-binary-copy\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961597    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-kubelet\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-systemd-units\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961671    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-run-systemd\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961693    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-var-lib-openvswitch\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-etc-openvswitch\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-cni-netd\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-env-overrides\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d50d211-9d46-4e14-ace4-554592a1ddc0-host\") pod \"node-ca-kd9fr\" (UID: \"6d50d211-9d46-4e14-ace4-554592a1ddc0\") " pod="openshift-image-registry/node-ca-kd9fr"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-os-release\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d50d211-9d46-4e14-ace4-554592a1ddc0-host\") pod \"node-ca-kd9fr\" (UID: \"6d50d211-9d46-4e14-ace4-554592a1ddc0\") " pod="openshift-image-registry/node-ca-kd9fr"
Jul 24 00:13:42.145436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-var-lib-cni-multus\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961931    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-var-lib-cni-multus\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-socket-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961970    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bd3079-c0b1-4319-9e19-566a47f07873-tmp\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.961995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-ovnkube-config\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-cni-dir\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962045    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-socket-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962065    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-os-release\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962068    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-device-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-cnibin\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-daemon-config\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-cnibin\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962277    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-cni-binary-copy\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71f46a43-3c16-45f8-90c4-0f815e6850d0-dbus\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-cni-dir\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.147536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962113    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-device-dir\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962384    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-run-openvswitch\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-lib-modules\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962444    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-var-lib-kubelet\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962452    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b009975-4390-46dd-9c1a-609bb499b2c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962517    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-run-k8s-cni-cncf-io\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-run-k8s-cni-cncf-io\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-run-netns\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962675    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-cni-bin\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-ovnkube-script-lib\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962766    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-multus-daemon-config\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-host-run-netns\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b87ed47a-6f1c-4f68-ae51-11144c46d320-konnectivity-ca\") pod \"konnectivity-agent-xctk4\" (UID: \"b87ed47a-6f1c-4f68-ae51-11144c46d320\") " pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.962974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.963427    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b87ed47a-6f1c-4f68-ae51-11144c46d320-konnectivity-ca\") pod \"konnectivity-agent-xctk4\" (UID: \"b87ed47a-6f1c-4f68-ae51-11144c46d320\") " pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:13:42.148449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.966019    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b87ed47a-6f1c-4f68-ae51-11144c46d320-agent-certs\") pod \"konnectivity-agent-xctk4\" (UID: \"b87ed47a-6f1c-4f68-ae51-11144c46d320\") " pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.966565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" event={"ID":"1c416aa043ef3695f72f46e3bfb88066","Type":"ContainerStarted","Data":"46c4efd850f768509fda593da70de9974a74f53b6e3359a42c727df87b53f7e1"}
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.968393    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" event={"ID":"18fde8ac8173b64a16e5b7537a21972f","Type":"ContainerStarted","Data":"0ed06cd03e16522ec4d9691996074faf158b6a5fdd1ef2296a47b9d349079169"}
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.968723    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dgt4\" (UniqueName: \"kubernetes.io/projected/6d50d211-9d46-4e14-ace4-554592a1ddc0-kube-api-access-9dgt4\") pod \"node-ca-kd9fr\" (UID: \"6d50d211-9d46-4e14-ace4-554592a1ddc0\") " pod="openshift-image-registry/node-ca-kd9fr"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.969172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dprz\" (UniqueName: \"kubernetes.io/projected/4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8-kube-api-access-6dprz\") pod \"multus-additional-cni-plugins-kw8qh\" (UID: \"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8\") " pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.969705    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lsq2\" (UniqueName: \"kubernetes.io/projected/7b009975-4390-46dd-9c1a-609bb499b2c3-kube-api-access-9lsq2\") pod \"aws-ebs-csi-driver-node-wk792\" (UID: \"7b009975-4390-46dd-9c1a-609bb499b2c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.969722    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2zs2\" (UniqueName: \"kubernetes.io/projected/2a888e7d-1f98-4081-a014-cca49462de1f-kube-api-access-x2zs2\") pod \"iptables-alerter-66t22\" (UID: \"2a888e7d-1f98-4081-a014-cca49462de1f\") " pod="openshift-network-operator/iptables-alerter-66t22"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:41.972005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7j6d\" (UniqueName: \"kubernetes.io/projected/117554d8-b8ea-4604-ad9f-9ad18d90a7f9-kube-api-access-s7j6d\") pod \"multus-zjgvk\" (UID: \"117554d8-b8ea-4604-ad9f-9ad18d90a7f9\") " pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-ovnkube-script-lib\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-sysctl-d\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-node-log\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-sysconfig\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063320    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71f46a43-3c16-45f8-90c4-0f815e6850d0-kubelet-config\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-ovn-node-metrics-cert\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063365    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwcx7\" (UniqueName: \"kubernetes.io/projected/55bd3079-c0b1-4319-9e19-566a47f07873-kube-api-access-lwcx7\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.063385    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063359    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-node-log\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.149851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063417    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71f46a43-3c16-45f8-90c4-0f815e6850d0-kubelet-config\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-slash\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063442    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-sysctl-d\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063458    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-sysconfig\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.063470    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs podName:b63b995d-5404-4953-acf6-31e068d8cd35 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:42.5634449 +0000 UTC m=+3.131261054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs") pod "network-metrics-daemon-7kps9" (UID: "b63b995d-5404-4953-acf6-31e068d8cd35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-slash\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063628    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-sysctl-conf\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-modprobe-d\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063610    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063767    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-run\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-sysctl-conf\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-log-socket\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063826    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-run\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-systemd\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063882    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a00cf2c5-5c20-410f-bae2-80170581b317-hosts-file\") pod \"node-resolver-mttpm\" (UID: \"a00cf2c5-5c20-410f-bae2-80170581b317\") " pod="openshift-dns/node-resolver-mttpm"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063888    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-log-socket\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.150787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-modprobe-d\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063909    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-ovnkube-script-lib\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063938    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a00cf2c5-5c20-410f-bae2-80170581b317-hosts-file\") pod \"node-resolver-mttpm\" (UID: \"a00cf2c5-5c20-410f-bae2-80170581b317\") " pod="openshift-dns/node-resolver-mttpm"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-systemd\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.063995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a00cf2c5-5c20-410f-bae2-80170581b317-tmp-dir\") pod \"node-resolver-mttpm\" (UID: \"a00cf2c5-5c20-410f-bae2-80170581b317\") " pod="openshift-dns/node-resolver-mttpm"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-run-ovn\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064066    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-host\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5xhk\" (UniqueName: \"kubernetes.io/projected/a00cf2c5-5c20-410f-bae2-80170581b317-kube-api-access-b5xhk\") pod \"node-resolver-mttpm\" (UID: \"a00cf2c5-5c20-410f-bae2-80170581b317\") " pod="openshift-dns/node-resolver-mttpm"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-run-ovn\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064147    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xb6\" (UniqueName: \"kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6\") pod \"network-check-target-r722z\" (UID: \"1bae97a1-6b3a-4ff1-847b-2df09e067715\") " pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064145    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064157    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-host\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064254    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a00cf2c5-5c20-410f-bae2-80170581b317-tmp-dir\") pod \"node-resolver-mttpm\" (UID: \"a00cf2c5-5c20-410f-bae2-80170581b317\") " pod="openshift-dns/node-resolver-mttpm"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fkw\" (UniqueName: \"kubernetes.io/projected/b63b995d-5404-4953-acf6-31e068d8cd35-kube-api-access-v9fkw\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-kubernetes\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.064378    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:42.152063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064416    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-run-netns\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-run-netns\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.064446    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret podName:71f46a43-3c16-45f8-90c4-0f815e6850d0 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:42.564430095 +0000 UTC m=+3.132246237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret") pod "global-pull-secret-syncer-kgsn8" (UID: "71f46a43-3c16-45f8-90c4-0f815e6850d0") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064459    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-etc-kubernetes\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t44gp\" (UniqueName: \"kubernetes.io/projected/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-kube-api-access-t44gp\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-sys\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55bd3079-c0b1-4319-9e19-566a47f07873-etc-tuned\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-kubelet\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064567    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-systemd-units\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-run-systemd\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-var-lib-openvswitch\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064618    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-kubelet\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064631    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-systemd-units\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-etc-openvswitch\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-sys\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-cni-netd\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-var-lib-openvswitch\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.152942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064691    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-env-overrides\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-etc-openvswitch\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-run-systemd\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-cni-netd\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064740    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bd3079-c0b1-4319-9e19-566a47f07873-tmp\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-ovnkube-config\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064812    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71f46a43-3c16-45f8-90c4-0f815e6850d0-dbus\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-run-openvswitch\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-lib-modules\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-var-lib-kubelet\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.064955    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-cni-bin\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.065027    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-host-cni-bin\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.065074    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-run-openvswitch\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.065109    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-env-overrides\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.065140    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-var-lib-kubelet\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.065175    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71f46a43-3c16-45f8-90c4-0f815e6850d0-dbus\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.065223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55bd3079-c0b1-4319-9e19-566a47f07873-lib-modules\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.066051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-ovnkube-config\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.153853 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.066319    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-ovn-node-metrics-cert\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.155452 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.067190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bd3079-c0b1-4319-9e19-566a47f07873-tmp\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.155452 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.068143    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55bd3079-c0b1-4319-9e19-566a47f07873-etc-tuned\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.155452 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.069775    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:13:42.155452 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.069795    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:13:42.155452 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.069805    2560 projected.go:196] Error preparing data for projected volume kube-api-access-r2xb6 for pod openshift-network-diagnostics/network-check-target-r722z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:42.155452 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.069870    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6 podName:1bae97a1-6b3a-4ff1-847b-2df09e067715 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:42.56985885 +0000 UTC m=+3.137675007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r2xb6" (UniqueName: "kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6") pod "network-check-target-r722z" (UID: "1bae97a1-6b3a-4ff1-847b-2df09e067715") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:42.155452 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.072668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwcx7\" (UniqueName: \"kubernetes.io/projected/55bd3079-c0b1-4319-9e19-566a47f07873-kube-api-access-lwcx7\") pod \"tuned-fz6zv\" (UID: \"55bd3079-c0b1-4319-9e19-566a47f07873\") " pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.155452 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.072953    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5xhk\" (UniqueName: \"kubernetes.io/projected/a00cf2c5-5c20-410f-bae2-80170581b317-kube-api-access-b5xhk\") pod \"node-resolver-mttpm\" (UID: \"a00cf2c5-5c20-410f-bae2-80170581b317\") " pod="openshift-dns/node-resolver-mttpm"
Jul 24 00:13:42.155452 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.073348    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fkw\" (UniqueName: \"kubernetes.io/projected/b63b995d-5404-4953-acf6-31e068d8cd35-kube-api-access-v9fkw\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:42.155452 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.075768    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t44gp\" (UniqueName: \"kubernetes.io/projected/a2af8c08-c07f-4233-acc0-2011d8f0a7e0-kube-api-access-t44gp\") pod \"ovnkube-node-62qhx\" (UID: \"a2af8c08-c07f-4233-acc0-2011d8f0a7e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.155452 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.150737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-66t22"
Jul 24 00:13:42.229647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.158014    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792"
Jul 24 00:13:42.229647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.162188    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kd9fr"
Jul 24 00:13:42.229647 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:42.167010    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a888e7d_1f98_4081_a014_cca49462de1f.slice/crio-ab48f20245fb94be996293d0a90f9ada26af3c515954e9afa46274b8dfe4bc32 WatchSource:0}: Error finding container ab48f20245fb94be996293d0a90f9ada26af3c515954e9afa46274b8dfe4bc32: Status 404 returned error can't find the container with id ab48f20245fb94be996293d0a90f9ada26af3c515954e9afa46274b8dfe4bc32
Jul 24 00:13:42.229647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.172906    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:13:42.229647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.178384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kw8qh"
Jul 24 00:13:42.229647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.188319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zjgvk"
Jul 24 00:13:42.229647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.194651    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:13:42.229647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.202197    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fz6zv"
Jul 24 00:13:42.229647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.206353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mttpm"
Jul 24 00:13:42.372534 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.322470    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:13:42.372534 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:42.328873    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf6f39f_3f1e_40d4_b5cc_2937ed06c2e8.slice/crio-0225b4c3a3e6dad0cbc774a45a6165cea2c3759f4bdea5d750d03a1964996e88 WatchSource:0}: Error finding container 0225b4c3a3e6dad0cbc774a45a6165cea2c3759f4bdea5d750d03a1964996e88: Status 404 returned error can't find the container with id 0225b4c3a3e6dad0cbc774a45a6165cea2c3759f4bdea5d750d03a1964996e88
Jul 24 00:13:42.372534 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:42.329463    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87ed47a_6f1c_4f68_ae51_11144c46d320.slice/crio-30be947462c9868d8215a306d7fe321f8fd3b4c8cdb0e68e3e69a91d5ecf1e58 WatchSource:0}: Error finding container 30be947462c9868d8215a306d7fe321f8fd3b4c8cdb0e68e3e69a91d5ecf1e58: Status 404 returned error can't find the container with id 30be947462c9868d8215a306d7fe321f8fd3b4c8cdb0e68e3e69a91d5ecf1e58
Jul 24 00:13:42.372534 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:42.337705    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117554d8_b8ea_4604_ad9f_9ad18d90a7f9.slice/crio-adba2439d2ff69b60450ec0070664a80fd6ad6a2f8977e2556a9c6d4de7141ce WatchSource:0}: Error finding container adba2439d2ff69b60450ec0070664a80fd6ad6a2f8977e2556a9c6d4de7141ce: Status 404 returned error can't find the container with id adba2439d2ff69b60450ec0070664a80fd6ad6a2f8977e2556a9c6d4de7141ce
Jul 24 00:13:42.372534 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:13:42.338373    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00cf2c5_5c20_410f_bae2_80170581b317.slice/crio-0976cd99d36c3949a564962505f1a690cf8272583036fc92b758b4557b6f25a0 WatchSource:0}: Error finding container 0976cd99d36c3949a564962505f1a690cf8272583036fc92b758b4557b6f25a0: Status 404 returned error can't find the container with id 0976cd99d36c3949a564962505f1a690cf8272583036fc92b758b4557b6f25a0
Jul 24 00:13:42.700655 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.569759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:42.700655 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.569809    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:42.700655 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.569984    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:42.700655 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.570078    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret podName:71f46a43-3c16-45f8-90c4-0f815e6850d0 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:43.57005444 +0000 UTC m=+4.137870587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret") pod "global-pull-secret-syncer-kgsn8" (UID: "71f46a43-3c16-45f8-90c4-0f815e6850d0") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:42.700655 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.569984    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:42.700655 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.570159    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs podName:b63b995d-5404-4953-acf6-31e068d8cd35 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:43.570139528 +0000 UTC m=+4.137955680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs") pod "network-metrics-daemon-7kps9" (UID: "b63b995d-5404-4953-acf6-31e068d8cd35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:42.700655 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.670428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xb6\" (UniqueName: \"kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6\") pod \"network-check-target-r722z\" (UID: \"1bae97a1-6b3a-4ff1-847b-2df09e067715\") " pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:42.700655 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.670642    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:13:42.700655 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.670667    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:13:42.700655 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.670681    2560 projected.go:196] Error preparing data for projected volume kube-api-access-r2xb6 for pod openshift-network-diagnostics/network-check-target-r722z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:42.700655 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:42.670759    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6 podName:1bae97a1-6b3a-4ff1-847b-2df09e067715 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:43.670735449 +0000 UTC m=+4.238551580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2xb6" (UniqueName: "kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6") pod "network-check-target-r722z" (UID: "1bae97a1-6b3a-4ff1-847b-2df09e067715") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:42.978847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.898704    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 00:08:42 +0000 UTC" deadline="2028-04-25 00:05:28.14763274 +0000 UTC"
Jul 24 00:13:42.978847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.898743    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15383h51m45.2488945s"
Jul 24 00:13:42.978847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.972615    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fz6zv" event={"ID":"55bd3079-c0b1-4319-9e19-566a47f07873","Type":"ContainerStarted","Data":"7d422aa5387e6cce0ddcc2c657c3ae2dc1229ec21c7440d9b8c09ba6119d0b87"}
Jul 24 00:13:42.978847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.974069    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mttpm" event={"ID":"a00cf2c5-5c20-410f-bae2-80170581b317","Type":"ContainerStarted","Data":"0976cd99d36c3949a564962505f1a690cf8272583036fc92b758b4557b6f25a0"}
Jul 24 00:13:42.978847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.975397    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kw8qh" event={"ID":"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8","Type":"ContainerStarted","Data":"0225b4c3a3e6dad0cbc774a45a6165cea2c3759f4bdea5d750d03a1964996e88"}
Jul 24 00:13:42.978847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.976804    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx" event={"ID":"a2af8c08-c07f-4233-acc0-2011d8f0a7e0","Type":"ContainerStarted","Data":"928a3eac2ca6f7610b5d4eb073fcb8bfb6b5b2d11c246607d3051426dc44fd52"}
Jul 24 00:13:42.978847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.978153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zjgvk" event={"ID":"117554d8-b8ea-4604-ad9f-9ad18d90a7f9","Type":"ContainerStarted","Data":"adba2439d2ff69b60450ec0070664a80fd6ad6a2f8977e2556a9c6d4de7141ce"}
Jul 24 00:13:42.980260 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.979368    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xctk4" event={"ID":"b87ed47a-6f1c-4f68-ae51-11144c46d320","Type":"ContainerStarted","Data":"30be947462c9868d8215a306d7fe321f8fd3b4c8cdb0e68e3e69a91d5ecf1e58"}
Jul 24 00:13:42.981644 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.980724    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kd9fr" event={"ID":"6d50d211-9d46-4e14-ace4-554592a1ddc0","Type":"ContainerStarted","Data":"5fa04a65826e89de3f9eb1d5b6758f28b40c8e1aae5668807808dde8a8c930ab"}
Jul 24 00:13:42.982880 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.982049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792" event={"ID":"7b009975-4390-46dd-9c1a-609bb499b2c3","Type":"ContainerStarted","Data":"bd47b6720505cb2719f739f2b792a7c668b8a4a79479f774c04765d3d1d2a64e"}
Jul 24 00:13:42.983304 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:42.983280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-66t22" event={"ID":"2a888e7d-1f98-4081-a014-cca49462de1f","Type":"ContainerStarted","Data":"ab48f20245fb94be996293d0a90f9ada26af3c515954e9afa46274b8dfe4bc32"}
Jul 24 00:13:43.577318 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:43.577170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:43.577318 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:43.577260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:43.577934 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:43.577450    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:43.577934 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:43.577536    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs podName:b63b995d-5404-4953-acf6-31e068d8cd35 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:45.577513627 +0000 UTC m=+6.145329762 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs") pod "network-metrics-daemon-7kps9" (UID: "b63b995d-5404-4953-acf6-31e068d8cd35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:43.578034 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:43.577992    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:43.578067 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:43.578049    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret podName:71f46a43-3c16-45f8-90c4-0f815e6850d0 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:45.578031997 +0000 UTC m=+6.145848201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret") pod "global-pull-secret-syncer-kgsn8" (UID: "71f46a43-3c16-45f8-90c4-0f815e6850d0") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:43.678258 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:43.678219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xb6\" (UniqueName: \"kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6\") pod \"network-check-target-r722z\" (UID: \"1bae97a1-6b3a-4ff1-847b-2df09e067715\") " pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:43.678427 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:43.678401    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:13:43.678427 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:43.678422    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:13:43.678557 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:43.678437    2560 projected.go:196] Error preparing data for projected volume kube-api-access-r2xb6 for pod openshift-network-diagnostics/network-check-target-r722z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:43.678557 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:43.678516    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6 podName:1bae97a1-6b3a-4ff1-847b-2df09e067715 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:45.678496555 +0000 UTC m=+6.246312700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2xb6" (UniqueName: "kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6") pod "network-check-target-r722z" (UID: "1bae97a1-6b3a-4ff1-847b-2df09e067715") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:43.899895 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:43.899800    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 00:08:42 +0000 UTC" deadline="2028-02-22 03:11:10.496002354 +0000 UTC"
Jul 24 00:13:43.899895 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:43.899864    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13874h57m26.596143119s"
Jul 24 00:13:43.950479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:43.950120    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 00:13:43.962015 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:43.961988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:43.962152 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:43.962127    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:13:43.962584 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:43.962555    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:43.962701 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:43.962677    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:13:43.962872 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:43.962854    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:43.962990 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:43.962961    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:13:45.002954 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:45.002513    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" event={"ID":"1c416aa043ef3695f72f46e3bfb88066","Type":"ContainerStarted","Data":"09e7b455855e0836fa3d7ac68caaea2183a6abfc8cf5c2fefb0ca676051e7419"}
Jul 24 00:13:45.013809 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:45.013262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" event={"ID":"18fde8ac8173b64a16e5b7537a21972f","Type":"ContainerStarted","Data":"137de1124fb4ce84e54ff0997d388465a55a8bed5b1c0456e485d3bb5f0732bf"}
Jul 24 00:13:45.036652 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:45.036395    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-176.ec2.internal" podStartSLOduration=4.036371973 podStartE2EDuration="4.036371973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:13:45.035328211 +0000 UTC m=+5.603144378" watchObservedRunningTime="2026-07-24 00:13:45.036371973 +0000 UTC m=+5.604188129"
Jul 24 00:13:45.597430 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:45.596464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:45.597430 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:45.596541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:45.597430 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:45.596693    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:45.597430 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:45.596760    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs podName:b63b995d-5404-4953-acf6-31e068d8cd35 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:49.596739138 +0000 UTC m=+10.164555273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs") pod "network-metrics-daemon-7kps9" (UID: "b63b995d-5404-4953-acf6-31e068d8cd35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:45.597430 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:45.597240    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:45.597430 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:45.597311    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret podName:71f46a43-3c16-45f8-90c4-0f815e6850d0 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:49.597295124 +0000 UTC m=+10.165111278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret") pod "global-pull-secret-syncer-kgsn8" (UID: "71f46a43-3c16-45f8-90c4-0f815e6850d0") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:45.697365 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:45.697264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xb6\" (UniqueName: \"kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6\") pod \"network-check-target-r722z\" (UID: \"1bae97a1-6b3a-4ff1-847b-2df09e067715\") " pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:45.697704 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:45.697526    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:13:45.697704 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:45.697552    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:13:45.697704 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:45.697565    2560 projected.go:196] Error preparing data for projected volume kube-api-access-r2xb6 for pod openshift-network-diagnostics/network-check-target-r722z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:45.697704 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:45.697624    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6 podName:1bae97a1-6b3a-4ff1-847b-2df09e067715 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:49.697605424 +0000 UTC m=+10.265421580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2xb6" (UniqueName: "kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6") pod "network-check-target-r722z" (UID: "1bae97a1-6b3a-4ff1-847b-2df09e067715") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:45.962054 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:45.962020    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:45.962243 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:45.962157    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:45.962243 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:45.962178    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:13:45.962367 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:45.962273    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:13:45.962367 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:45.962019    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:45.962501 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:45.962457    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:13:47.961147 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:47.961115    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:47.961584 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:47.961119    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:47.961584 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:47.961247    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:13:47.961584 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:47.961270    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:47.961584 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:47.961375    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:13:47.961584 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:47.961459    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:13:49.632613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:49.632285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:49.632613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:49.632397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:49.632613 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:49.632459    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:49.632613 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:49.632532    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:49.632613 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:49.632543    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs podName:b63b995d-5404-4953-acf6-31e068d8cd35 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:57.632519804 +0000 UTC m=+18.200335934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs") pod "network-metrics-daemon-7kps9" (UID: "b63b995d-5404-4953-acf6-31e068d8cd35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:49.632613 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:49.632587    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret podName:71f46a43-3c16-45f8-90c4-0f815e6850d0 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:57.63256956 +0000 UTC m=+18.200385701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret") pod "global-pull-secret-syncer-kgsn8" (UID: "71f46a43-3c16-45f8-90c4-0f815e6850d0") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:49.733159 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:49.733123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xb6\" (UniqueName: \"kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6\") pod \"network-check-target-r722z\" (UID: \"1bae97a1-6b3a-4ff1-847b-2df09e067715\") " pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:49.733334 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:49.733309    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:13:49.733414 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:49.733342    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:13:49.733414 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:49.733358    2560 projected.go:196] Error preparing data for projected volume kube-api-access-r2xb6 for pod openshift-network-diagnostics/network-check-target-r722z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:49.733525 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:49.733426    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6 podName:1bae97a1-6b3a-4ff1-847b-2df09e067715 nodeName:}" failed. No retries permitted until 2026-07-24 00:13:57.733405273 +0000 UTC m=+18.301221417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2xb6" (UniqueName: "kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6") pod "network-check-target-r722z" (UID: "1bae97a1-6b3a-4ff1-847b-2df09e067715") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:49.962569 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:49.962216    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:49.962569 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:49.962308    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:49.962569 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:49.962362    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:49.962569 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:49.962355    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:13:49.962569 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:49.962458    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:13:49.962569 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:49.962533    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:13:50.027165 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:50.027130    2560 generic.go:356] "Generic (PLEG): container finished" podID="1c416aa043ef3695f72f46e3bfb88066" containerID="09e7b455855e0836fa3d7ac68caaea2183a6abfc8cf5c2fefb0ca676051e7419" exitCode=0
Jul 24 00:13:50.027351 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:50.027166    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" event={"ID":"1c416aa043ef3695f72f46e3bfb88066","Type":"ContainerDied","Data":"09e7b455855e0836fa3d7ac68caaea2183a6abfc8cf5c2fefb0ca676051e7419"}
Jul 24 00:13:51.961275 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:51.961231    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:51.961275 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:51.961269    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:51.961790 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:51.961231    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:51.961790 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:51.961400    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:13:51.961790 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:51.961486    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:13:51.961790 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:51.961574    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:13:53.965725 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:53.965698    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:53.966168 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:53.965808    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:13:53.966168 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:53.965699    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:53.966168 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:53.965697    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:53.966168 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:53.965907    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:13:53.966168 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:53.965978    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:13:55.962079 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:55.962039    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:55.962498 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:55.962039    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:55.962498 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:55.962166    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:13:55.962498 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:55.962274    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:13:55.962498 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:55.962051    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:55.962498 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:55.962378    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:13:57.691496 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:57.691460    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:57.691922 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:57.691520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:57.691922 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:57.691604    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:57.691922 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:57.691617    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:57.691922 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:57.691662    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret podName:71f46a43-3c16-45f8-90c4-0f815e6850d0 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:13.691647986 +0000 UTC m=+34.259464118 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret") pod "global-pull-secret-syncer-kgsn8" (UID: "71f46a43-3c16-45f8-90c4-0f815e6850d0") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:13:57.691922 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:57.691678    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs podName:b63b995d-5404-4953-acf6-31e068d8cd35 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:13.691670947 +0000 UTC m=+34.259487080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs") pod "network-metrics-daemon-7kps9" (UID: "b63b995d-5404-4953-acf6-31e068d8cd35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:13:57.792884 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:57.792845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xb6\" (UniqueName: \"kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6\") pod \"network-check-target-r722z\" (UID: \"1bae97a1-6b3a-4ff1-847b-2df09e067715\") " pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:57.793049 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:57.793003    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:13:57.793049 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:57.793023    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:13:57.793049 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:57.793033    2560 projected.go:196] Error preparing data for projected volume kube-api-access-r2xb6 for pod openshift-network-diagnostics/network-check-target-r722z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:57.793211 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:57.793091    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6 podName:1bae97a1-6b3a-4ff1-847b-2df09e067715 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:13.793074432 +0000 UTC m=+34.360890579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2xb6" (UniqueName: "kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6") pod "network-check-target-r722z" (UID: "1bae97a1-6b3a-4ff1-847b-2df09e067715") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:13:57.961702 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:57.961630    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:57.961702 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:57.961669    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:57.961948 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:57.961630    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:57.961948 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:57.961756    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:13:57.961948 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:57.961853    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:13:57.962099 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:57.961956    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:13:59.962422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:59.962382    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:13:59.962891 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:59.962490    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:13:59.962891 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:13:59.962544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:13:59.962891 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:59.962550    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:13:59.962891 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:59.962621    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:13:59.962891 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:13:59.962701    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:14:01.961577 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:01.961288    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:01.961577 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:01.961417    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:14:01.961577 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:01.961478    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:01.961577 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:01.961543    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:14:01.962476 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:01.961954    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:01.962476 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:01.962051    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:14:02.054948 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.054745    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mttpm" event={"ID":"a00cf2c5-5c20-410f-bae2-80170581b317","Type":"ContainerStarted","Data":"4b06318073d7557f3b3d4f36ac51f4f8cfaf0419114266362c240ab3f18bbb8a"}
Jul 24 00:14:02.057348 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.057159    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kw8qh" event={"ID":"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8","Type":"ContainerStarted","Data":"1e9c01d331df5ef7bd97ac2b571e10957b144ced2fcbb10d97400dd2cb9703b6"}
Jul 24 00:14:02.061026 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.060755    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zjgvk" event={"ID":"117554d8-b8ea-4604-ad9f-9ad18d90a7f9","Type":"ContainerStarted","Data":"9dd3ffe89fbc0347081c781593e70ac85a3ffe3b56f3edd7376bc65a7b11f4a4"}
Jul 24 00:14:02.065347 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.064873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xctk4" event={"ID":"b87ed47a-6f1c-4f68-ae51-11144c46d320","Type":"ContainerStarted","Data":"b6c0f18c805d42d12b77b8099363d529e489a80f2d3082c9a5787cbdcecd879c"}
Jul 24 00:14:02.071266 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.071233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792" event={"ID":"7b009975-4390-46dd-9c1a-609bb499b2c3","Type":"ContainerStarted","Data":"7bbfd5dd5100ff1388371c98beb17c407ebec90b1d953491a895542a72ee8a35"}
Jul 24 00:14:02.071722 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.071651    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mttpm" podStartSLOduration=3.066346994 podStartE2EDuration="22.071631712s" totalImagesPullingTime="19.005284718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:02.070815707 +0000 UTC m=+22.638631869" watchObservedRunningTime="2026-07-24 00:14:02.071631712 +0000 UTC m=+22.639447862"
Jul 24 00:14:02.073141 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.073100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" event={"ID":"1c416aa043ef3695f72f46e3bfb88066","Type":"ContainerStarted","Data":"297a4cddc44131362296a7f704c6f20951e3429b155e40683a8ec8d90b5fb062"}
Jul 24 00:14:02.074632 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.074610    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fz6zv" event={"ID":"55bd3079-c0b1-4319-9e19-566a47f07873","Type":"ContainerStarted","Data":"ff44569e0b3e66e5ce6cc908a31035eac2fae430c23b683b07bbef2a077febac"}
Jul 24 00:14:02.098275 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.098231    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xctk4" podStartSLOduration=2.711677708 podStartE2EDuration="22.098214361s" totalImagesPullingTime="19.386536653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:02.097483101 +0000 UTC m=+22.665299251" watchObservedRunningTime="2026-07-24 00:14:02.098214361 +0000 UTC m=+22.666030512"
Jul 24 00:14:02.098471 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.098442    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kd9fr" podStartSLOduration=2.671133573 podStartE2EDuration="22.098432322s" totalImagesPullingTime="19.427298749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:02.084549359 +0000 UTC m=+22.652365508" watchObservedRunningTime="2026-07-24 00:14:02.098432322 +0000 UTC m=+22.666248472"
Jul 24 00:14:02.142641 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.142583    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zjgvk" podStartSLOduration=2.733096766 podStartE2EDuration="22.14256683s" totalImagesPullingTime="19.409470064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:02.142467283 +0000 UTC m=+22.710283434" watchObservedRunningTime="2026-07-24 00:14:02.14256683 +0000 UTC m=+22.710382980"
Jul 24 00:14:02.157769 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.157583    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-176.ec2.internal" podStartSLOduration=21.157568898 podStartE2EDuration="21.157568898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:02.157089021 +0000 UTC m=+22.724905162" watchObservedRunningTime="2026-07-24 00:14:02.157568898 +0000 UTC m=+22.725385047"
Jul 24 00:14:02.174883 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:02.174802    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fz6zv" podStartSLOduration=2.770260683 podStartE2EDuration="22.174780238s" totalImagesPullingTime="19.404519555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:02.174723549 +0000 UTC m=+22.742539700" watchObservedRunningTime="2026-07-24 00:14:02.174780238 +0000 UTC m=+22.742596390"
Jul 24 00:14:03.080155 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.080121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx" event={"ID":"a2af8c08-c07f-4233-acc0-2011d8f0a7e0","Type":"ContainerStarted","Data":"6f8b2d934d37173a4d78bd9f30452d6d7f8fa2be73edd0c27b6a6bdb0062b5a5"}
Jul 24 00:14:03.080847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.080218    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx" event={"ID":"a2af8c08-c07f-4233-acc0-2011d8f0a7e0","Type":"ContainerStarted","Data":"dc6889cda52d77ac289a9bc5d747d0d41c8d715a148973b3526e9ae3544ccd0c"}
Jul 24 00:14:03.080847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.080249    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx" event={"ID":"a2af8c08-c07f-4233-acc0-2011d8f0a7e0","Type":"ContainerStarted","Data":"36a1ac181b8f3fb0b749748996910b802ba22550fc749b01cd6dbff4804b15a1"}
Jul 24 00:14:03.080847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.080278    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx" event={"ID":"a2af8c08-c07f-4233-acc0-2011d8f0a7e0","Type":"ContainerStarted","Data":"3c890187b1bf9444b8edb9551ae4ed3387bfb612cd88cc98180520da4a574c83"}
Jul 24 00:14:03.080847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.080303    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx" event={"ID":"a2af8c08-c07f-4233-acc0-2011d8f0a7e0","Type":"ContainerStarted","Data":"d65065e06d8bd1a56e92adde040f3b2467605e9129cfd26a85c3a076b932e0d0"}
Jul 24 00:14:03.080847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.080329    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx" event={"ID":"a2af8c08-c07f-4233-acc0-2011d8f0a7e0","Type":"ContainerStarted","Data":"4885ccc40be2c25ec59edadbaadb8f511728914f5160ce96f76241cdc6fa262f"}
Jul 24 00:14:03.081573 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.081552    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kd9fr" event={"ID":"6d50d211-9d46-4e14-ace4-554592a1ddc0","Type":"ContainerStarted","Data":"b389d12bf092cdaee1cdf79493e95d450966bce3da1b403de3794971fde09e7e"}
Jul 24 00:14:03.082886 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.082864    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-66t22" event={"ID":"2a888e7d-1f98-4081-a014-cca49462de1f","Type":"ContainerStarted","Data":"0a86d2a3c89b09e44c0f1e6bbb46599e3883ae3fd1b2ac88755bca96bf8cf657"}
Jul 24 00:14:03.084229 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.084209    2560 generic.go:356] "Generic (PLEG): container finished" podID="4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8" containerID="1e9c01d331df5ef7bd97ac2b571e10957b144ced2fcbb10d97400dd2cb9703b6" exitCode=0
Jul 24 00:14:03.084298 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.084278    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kw8qh" event={"ID":"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8","Type":"ContainerDied","Data":"1e9c01d331df5ef7bd97ac2b571e10957b144ced2fcbb10d97400dd2cb9703b6"}
Jul 24 00:14:03.106221 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.106185    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-66t22" podStartSLOduration=3.759240936 podStartE2EDuration="23.106172452s" totalImagesPullingTime="19.346931516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:03.105924046 +0000 UTC m=+23.673740195" watchObservedRunningTime="2026-07-24 00:14:03.106172452 +0000 UTC m=+23.673988602"
Jul 24 00:14:03.255190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.255165    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 00:14:03.769057 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.769023    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:14:03.769682 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.769660    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:14:03.923995 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.923899    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T00:14:03.255187307Z","UUID":"028cff42-8b3e-4d74-8538-4183f1ef439a","Handler":null,"Name":"","Endpoint":""}
Jul 24 00:14:03.926686 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.926665    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 00:14:03.926803 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.926701    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 00:14:03.962000 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.961970    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:03.962000 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.961992    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:03.962185 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:03.961970    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:03.962185 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:03.962073    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:14:03.962185 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:03.962134    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:14:03.962309 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:03.962215    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:14:04.089093 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:04.089010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792" event={"ID":"7b009975-4390-46dd-9c1a-609bb499b2c3","Type":"ContainerStarted","Data":"e3dd4737b47be0adf9f607dd09073ea119dc193fe65284620b04130b29f9f798"}
Jul 24 00:14:04.089522 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:04.089453    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:14:04.089934 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:04.089915    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xctk4"
Jul 24 00:14:05.094473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:05.094434    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx" event={"ID":"a2af8c08-c07f-4233-acc0-2011d8f0a7e0","Type":"ContainerStarted","Data":"585685c8f27e6aef39ed58a729272123328a3945f86dbda0acb71205dc10266d"}
Jul 24 00:14:05.096404 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:05.096370    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792" event={"ID":"7b009975-4390-46dd-9c1a-609bb499b2c3","Type":"ContainerStarted","Data":"8cc1a7330bb604c5497dc7bbe211234748bab754d7d41befd7a61810334d522a"}
Jul 24 00:14:05.114862 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:05.114794    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk792" podStartSLOduration=3.503143234 podStartE2EDuration="25.114775303s" totalImagesPullingTime="21.611632069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:05.113791469 +0000 UTC m=+25.681607620" watchObservedRunningTime="2026-07-24 00:14:05.114775303 +0000 UTC m=+25.682591454"
Jul 24 00:14:05.961931 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:05.961899    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:05.962115 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:05.961899    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:05.962115 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:05.962025    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:14:05.962115 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:05.962042    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:05.962441 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:05.962126    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:14:05.962441 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:05.962222    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:14:07.106265 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:07.105989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx" event={"ID":"a2af8c08-c07f-4233-acc0-2011d8f0a7e0","Type":"ContainerStarted","Data":"7e0d7a59cbb6934ddca6acbbd23256d22eb2b48283422a96429edebddc74e4fe"}
Jul 24 00:14:07.106808 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:07.106605    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:14:07.106808 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:07.106688    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:14:07.106808 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:07.106740    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:14:07.125173 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:07.125137    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:14:07.126258 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:07.126237    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:14:07.134590 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:07.134537    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx" podStartSLOduration=7.520224769 podStartE2EDuration="27.134519068s" totalImagesPullingTime="19.614294299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:07.133246316 +0000 UTC m=+27.701062467" watchObservedRunningTime="2026-07-24 00:14:07.134519068 +0000 UTC m=+27.702335229"
Jul 24 00:14:07.961690 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:07.961657    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:07.961898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:07.961657    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:07.961898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:07.961669    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:07.961898 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:07.961801    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:14:07.961898 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:07.961882    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:14:07.962067 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:07.961973    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:14:09.113149 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:09.112872    2560 generic.go:356] "Generic (PLEG): container finished" podID="4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8" containerID="61f96e5d13b5b811894b8dcae8e9414ba62009f7848a29e043b416f2c3a468ba" exitCode=0
Jul 24 00:14:09.113149 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:09.112952    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kw8qh" event={"ID":"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8","Type":"ContainerDied","Data":"61f96e5d13b5b811894b8dcae8e9414ba62009f7848a29e043b416f2c3a468ba"}
Jul 24 00:14:09.184030 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:09.184000    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kgsn8"]
Jul 24 00:14:09.184185 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:09.184104    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:09.184247 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:09.184202    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:14:09.187170 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:09.187143    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7kps9"]
Jul 24 00:14:09.187308 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:09.187260    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:09.187400 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:09.187381    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:14:09.188015 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:09.187989    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r722z"]
Jul 24 00:14:09.188123 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:09.188083    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:09.188199 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:09.188176    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:14:10.962018 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:10.961984    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:10.962018 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:10.962015    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:10.962402 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:10.962020    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:10.962402 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:10.962106    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:14:10.962402 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:10.962159    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:14:10.962402 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:10.962295    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:14:11.119037 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:11.119007    2560 generic.go:356] "Generic (PLEG): container finished" podID="4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8" containerID="a22df5c6e2c1d03bd79be52a313cd466891c6fbbf2b2629fe9b319f6d4185e1e" exitCode=0
Jul 24 00:14:11.119190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:11.119061    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kw8qh" event={"ID":"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8","Type":"ContainerDied","Data":"a22df5c6e2c1d03bd79be52a313cd466891c6fbbf2b2629fe9b319f6d4185e1e"}
Jul 24 00:14:12.962258 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:12.962068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:12.962609 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:12.962068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:12.962609 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:12.962353    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:14:12.962609 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:12.962068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:12.962609 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:12.962423    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:14:12.962609 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:12.962474    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:14:13.125764 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:13.125730    2560 generic.go:356] "Generic (PLEG): container finished" podID="4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8" containerID="17be38c664c7adb76884b1c1af3483d43390c66a16a9d2fb82c56c50b7dd4894" exitCode=0
Jul 24 00:14:13.125764 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:13.125769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kw8qh" event={"ID":"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8","Type":"ContainerDied","Data":"17be38c664c7adb76884b1c1af3483d43390c66a16a9d2fb82c56c50b7dd4894"}
Jul 24 00:14:13.713057 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:13.713022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:13.713221 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:13.713069    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:13.713221 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:13.713179    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 00:14:13.713320 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:13.713235    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:14:13.713320 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:13.713249    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret podName:71f46a43-3c16-45f8-90c4-0f815e6850d0 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:45.713228348 +0000 UTC m=+66.281044476 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret") pod "global-pull-secret-syncer-kgsn8" (UID: "71f46a43-3c16-45f8-90c4-0f815e6850d0") : object "kube-system"/"original-pull-secret" not registered
Jul 24 00:14:13.713320 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:13.713278    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs podName:b63b995d-5404-4953-acf6-31e068d8cd35 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:45.713266369 +0000 UTC m=+66.281082497 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs") pod "network-metrics-daemon-7kps9" (UID: "b63b995d-5404-4953-acf6-31e068d8cd35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 00:14:13.814189 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:13.814116    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xb6\" (UniqueName: \"kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6\") pod \"network-check-target-r722z\" (UID: \"1bae97a1-6b3a-4ff1-847b-2df09e067715\") " pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:13.814335 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:13.814309    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 00:14:13.814335 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:13.814331    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 00:14:13.814403 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:13.814344    2560 projected.go:196] Error preparing data for projected volume kube-api-access-r2xb6 for pod openshift-network-diagnostics/network-check-target-r722z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:14:13.814403 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:13.814400    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6 podName:1bae97a1-6b3a-4ff1-847b-2df09e067715 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:45.814385184 +0000 UTC m=+66.382201315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2xb6" (UniqueName: "kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6") pod "network-check-target-r722z" (UID: "1bae97a1-6b3a-4ff1-847b-2df09e067715") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 00:14:14.961744 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:14.961711    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:14.961744 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:14.961736    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:14.962260 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:14.961852    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kgsn8" podUID="71f46a43-3c16-45f8-90c4-0f815e6850d0"
Jul 24 00:14:14.962260 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:14.961898    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:14.962260 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:14.962034    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7kps9" podUID="b63b995d-5404-4953-acf6-31e068d8cd35"
Jul 24 00:14:14.962260 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:14.962121    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r722z" podUID="1bae97a1-6b3a-4ff1-847b-2df09e067715"
Jul 24 00:14:15.714632 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.714602    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-176.ec2.internal" event="NodeReady"
Jul 24 00:14:15.714936 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.714765    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 00:14:15.748764 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.748608    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-576d988ffd-2kxdz"]
Jul 24 00:14:15.774815 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.774788    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kfm7w"]
Jul 24 00:14:15.774999 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.774916    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:15.777554 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.777242    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 00:14:15.777554 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.777512    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 00:14:15.777768 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.777607    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qlxsm\""
Jul 24 00:14:15.777855 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.777817    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 00:14:15.781847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.781817    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 00:14:15.790522 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.790497    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7rbjj"]
Jul 24 00:14:15.790655 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.790637    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:15.792816 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.792749    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 00:14:15.792816 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.792759    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 00:14:15.792990 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.792899    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvjcq\""
Jul 24 00:14:15.808901 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.808878    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-576d988ffd-2kxdz"]
Jul 24 00:14:15.809023 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.808919    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kfm7w"]
Jul 24 00:14:15.809023 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.808945    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7rbjj"]
Jul 24 00:14:15.809023 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.808988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7rbjj"
Jul 24 00:14:15.810993 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.810952    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 00:14:15.811166 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.811044    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dwpqv\""
Jul 24 00:14:15.811240 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.811194    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 00:14:15.811298 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.811273    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 00:14:15.931154 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931092    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqxj8\" (UniqueName: \"kubernetes.io/projected/483be78f-e144-4a62-8932-88df4550a1af-kube-api-access-pqxj8\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:15.931418 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-installation-pull-secrets\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:15.931418 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac29741e-a8ad-4a01-8342-e631c2eca6ce-cert\") pod \"ingress-canary-7rbjj\" (UID: \"ac29741e-a8ad-4a01-8342-e631c2eca6ce\") " pod="openshift-ingress-canary/ingress-canary-7rbjj"
Jul 24 00:14:15.931418 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:15.931418 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-bound-sa-token\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:15.931418 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgx7h\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-kube-api-access-rgx7h\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:15.931418 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931358    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-image-registry-private-configuration\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:15.931418 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fa5df10-e395-47e4-a3d8-bea5b049a19f-ca-trust-extracted\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:15.931418 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931408    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:15.931933 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931454    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/483be78f-e144-4a62-8932-88df4550a1af-tmp-dir\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:15.931933 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931476    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-certificates\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:15.931933 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931495    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/483be78f-e144-4a62-8932-88df4550a1af-config-volume\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:15.931933 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqgn\" (UniqueName: \"kubernetes.io/projected/ac29741e-a8ad-4a01-8342-e631c2eca6ce-kube-api-access-rnqgn\") pod \"ingress-canary-7rbjj\" (UID: \"ac29741e-a8ad-4a01-8342-e631c2eca6ce\") " pod="openshift-ingress-canary/ingress-canary-7rbjj"
Jul 24 00:14:15.931933 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:15.931549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-trusted-ca\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.032947 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.032858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqxj8\" (UniqueName: \"kubernetes.io/projected/483be78f-e144-4a62-8932-88df4550a1af-kube-api-access-pqxj8\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:16.032947 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.032921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-installation-pull-secrets\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.032947 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.032944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac29741e-a8ad-4a01-8342-e631c2eca6ce-cert\") pod \"ingress-canary-7rbjj\" (UID: \"ac29741e-a8ad-4a01-8342-e631c2eca6ce\") " pod="openshift-ingress-canary/ingress-canary-7rbjj"
Jul 24 00:14:16.033485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.032964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.033485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.032987    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-bound-sa-token\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.033485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.033008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgx7h\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-kube-api-access-rgx7h\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.033485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.033049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-image-registry-private-configuration\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.033485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.033076    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fa5df10-e395-47e4-a3d8-bea5b049a19f-ca-trust-extracted\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.033485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.033101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:16.033485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.033147    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/483be78f-e144-4a62-8932-88df4550a1af-tmp-dir\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:16.033485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.033169    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-certificates\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.033485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.033194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/483be78f-e144-4a62-8932-88df4550a1af-config-volume\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:16.033485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.033215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqgn\" (UniqueName: \"kubernetes.io/projected/ac29741e-a8ad-4a01-8342-e631c2eca6ce-kube-api-access-rnqgn\") pod \"ingress-canary-7rbjj\" (UID: \"ac29741e-a8ad-4a01-8342-e631c2eca6ce\") " pod="openshift-ingress-canary/ingress-canary-7rbjj"
Jul 24 00:14:16.033989 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:16.033494    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 00:14:16.033989 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:16.033556    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls podName:483be78f-e144-4a62-8932-88df4550a1af nodeName:}" failed. No retries permitted until 2026-07-24 00:14:16.533537535 +0000 UTC m=+37.101353667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls") pod "dns-default-kfm7w" (UID: "483be78f-e144-4a62-8932-88df4550a1af") : secret "dns-default-metrics-tls" not found
Jul 24 00:14:16.033989 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.033560    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fa5df10-e395-47e4-a3d8-bea5b049a19f-ca-trust-extracted\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.033989 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.033612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-trusted-ca\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.033989 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:16.033870    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 00:14:16.033989 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:16.033886    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-576d988ffd-2kxdz: secret "image-registry-tls" not found
Jul 24 00:14:16.033989 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:16.033951    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls podName:1fa5df10-e395-47e4-a3d8-bea5b049a19f nodeName:}" failed. No retries permitted until 2026-07-24 00:14:16.533931644 +0000 UTC m=+37.101747788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls") pod "image-registry-576d988ffd-2kxdz" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f") : secret "image-registry-tls" not found
Jul 24 00:14:16.034361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.034112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/483be78f-e144-4a62-8932-88df4550a1af-tmp-dir\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:16.034592 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.034566    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-trusted-ca\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.034750 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.034676    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-certificates\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.035186 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.035162    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/483be78f-e144-4a62-8932-88df4550a1af-config-volume\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:16.038232 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.038204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-installation-pull-secrets\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.038232 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.038220    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-image-registry-private-configuration\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.038457 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.038239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac29741e-a8ad-4a01-8342-e631c2eca6ce-cert\") pod \"ingress-canary-7rbjj\" (UID: \"ac29741e-a8ad-4a01-8342-e631c2eca6ce\") " pod="openshift-ingress-canary/ingress-canary-7rbjj"
Jul 24 00:14:16.044243 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.044219    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqxj8\" (UniqueName: \"kubernetes.io/projected/483be78f-e144-4a62-8932-88df4550a1af-kube-api-access-pqxj8\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:16.044601 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.044567    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqgn\" (UniqueName: \"kubernetes.io/projected/ac29741e-a8ad-4a01-8342-e631c2eca6ce-kube-api-access-rnqgn\") pod \"ingress-canary-7rbjj\" (UID: \"ac29741e-a8ad-4a01-8342-e631c2eca6ce\") " pod="openshift-ingress-canary/ingress-canary-7rbjj"
Jul 24 00:14:16.044601 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.044581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgx7h\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-kube-api-access-rgx7h\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.045802 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.045775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-bound-sa-token\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.119036 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.119005    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7rbjj"
Jul 24 00:14:16.317491 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.317395    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7rbjj"]
Jul 24 00:14:16.321571 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:16.321544    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac29741e_a8ad_4a01_8342_e631c2eca6ce.slice/crio-1aaf250cd5836cf3e9d15c64ed9b173af4f835dab83b54a69a73391d0974398a WatchSource:0}: Error finding container 1aaf250cd5836cf3e9d15c64ed9b173af4f835dab83b54a69a73391d0974398a: Status 404 returned error can't find the container with id 1aaf250cd5836cf3e9d15c64ed9b173af4f835dab83b54a69a73391d0974398a
Jul 24 00:14:16.538793 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.538755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:16.538952 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.538800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:16.538952 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:16.538921    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 00:14:16.538952 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:16.538945    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-576d988ffd-2kxdz: secret "image-registry-tls" not found
Jul 24 00:14:16.539051 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:16.538997    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls podName:1fa5df10-e395-47e4-a3d8-bea5b049a19f nodeName:}" failed. No retries permitted until 2026-07-24 00:14:17.53898234 +0000 UTC m=+38.106798469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls") pod "image-registry-576d988ffd-2kxdz" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f") : secret "image-registry-tls" not found
Jul 24 00:14:16.539051 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:16.538919    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 00:14:16.539120 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:16.539080    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls podName:483be78f-e144-4a62-8932-88df4550a1af nodeName:}" failed. No retries permitted until 2026-07-24 00:14:17.539065794 +0000 UTC m=+38.106881925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls") pod "dns-default-kfm7w" (UID: "483be78f-e144-4a62-8932-88df4550a1af") : secret "dns-default-metrics-tls" not found
Jul 24 00:14:16.962099 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.961886    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:16.962306 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.961917    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:16.962306 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.961917    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:16.964709 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.964684    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 00:14:16.965299 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.965279    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 00:14:16.965447 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.965429    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 00:14:16.965510 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.965471    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jq7gh\""
Jul 24 00:14:16.965510 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.965500    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gpdt7\""
Jul 24 00:14:16.965614 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:16.965504    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 00:14:17.136066 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:17.136022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7rbjj" event={"ID":"ac29741e-a8ad-4a01-8342-e631c2eca6ce","Type":"ContainerStarted","Data":"1aaf250cd5836cf3e9d15c64ed9b173af4f835dab83b54a69a73391d0974398a"}
Jul 24 00:14:17.547763 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:17.547719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:17.547961 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:17.547780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:17.547961 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:17.547899    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 00:14:17.548056 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:17.547986    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls podName:483be78f-e144-4a62-8932-88df4550a1af nodeName:}" failed. No retries permitted until 2026-07-24 00:14:19.547968299 +0000 UTC m=+40.115784427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls") pod "dns-default-kfm7w" (UID: "483be78f-e144-4a62-8932-88df4550a1af") : secret "dns-default-metrics-tls" not found
Jul 24 00:14:17.548056 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:17.547902    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 00:14:17.548056 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:17.548011    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-576d988ffd-2kxdz: secret "image-registry-tls" not found
Jul 24 00:14:17.548056 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:17.548053    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls podName:1fa5df10-e395-47e4-a3d8-bea5b049a19f nodeName:}" failed. No retries permitted until 2026-07-24 00:14:19.548041429 +0000 UTC m=+40.115857569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls") pod "image-registry-576d988ffd-2kxdz" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f") : secret "image-registry-tls" not found
Jul 24 00:14:19.144287 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:19.144247    2560 generic.go:356] "Generic (PLEG): container finished" podID="4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8" containerID="10786f4677b30a1c691266363ad45888773151fa25fce264d493c5543bd4e9f2" exitCode=0
Jul 24 00:14:19.144693 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:19.144297    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kw8qh" event={"ID":"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8","Type":"ContainerDied","Data":"10786f4677b30a1c691266363ad45888773151fa25fce264d493c5543bd4e9f2"}
Jul 24 00:14:19.565112 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:19.565073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:19.565300 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:19.565163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:19.565300 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:19.565200    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 00:14:19.565300 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:19.565280    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls podName:483be78f-e144-4a62-8932-88df4550a1af nodeName:}" failed. No retries permitted until 2026-07-24 00:14:23.565259794 +0000 UTC m=+44.133075941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls") pod "dns-default-kfm7w" (UID: "483be78f-e144-4a62-8932-88df4550a1af") : secret "dns-default-metrics-tls" not found
Jul 24 00:14:19.565300 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:19.565282    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 00:14:19.565300 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:19.565296    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-576d988ffd-2kxdz: secret "image-registry-tls" not found
Jul 24 00:14:19.565552 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:19.565333    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls podName:1fa5df10-e395-47e4-a3d8-bea5b049a19f nodeName:}" failed. No retries permitted until 2026-07-24 00:14:23.565322904 +0000 UTC m=+44.133139031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls") pod "image-registry-576d988ffd-2kxdz" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f") : secret "image-registry-tls" not found
Jul 24 00:14:20.150202 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:20.150162    2560 generic.go:356] "Generic (PLEG): container finished" podID="4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8" containerID="2908ce8687062f2d874fcc6334147469ae8127fe016a42546441fadc2c3894b3" exitCode=0
Jul 24 00:14:20.150696 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:20.150219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kw8qh" event={"ID":"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8","Type":"ContainerDied","Data":"2908ce8687062f2d874fcc6334147469ae8127fe016a42546441fadc2c3894b3"}
Jul 24 00:14:21.155698 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:21.155660    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kw8qh" event={"ID":"4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8","Type":"ContainerStarted","Data":"cfaf8b39520b0d22a2e6959d6e0fe1cb9f0d173d2b8a86f0d02b9c3ac4b866d8"}
Jul 24 00:14:21.157032 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:21.157009    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7rbjj" event={"ID":"ac29741e-a8ad-4a01-8342-e631c2eca6ce","Type":"ContainerStarted","Data":"5cac96df123011d04a4bfd67df8f6773472f50f6290b104b26a3cf050cf15438"}
Jul 24 00:14:21.177904 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:21.177857    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kw8qh" podStartSLOduration=8.864421966 podStartE2EDuration="41.177823671s" totalImagesPullingTime="32.313401705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:21.176758839 +0000 UTC m=+41.744574989" watchObservedRunningTime="2026-07-24 00:14:21.177823671 +0000 UTC m=+41.745639822"
Jul 24 00:14:21.191111 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:21.191070    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7rbjj" podStartSLOduration=2.207995066 podStartE2EDuration="6.19106053s" totalImagesPullingTime="3.983065464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:21.190411294 +0000 UTC m=+41.758227443" watchObservedRunningTime="2026-07-24 00:14:21.19106053 +0000 UTC m=+41.758876718"
Jul 24 00:14:23.470497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.470451    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"]
Jul 24 00:14:23.494803 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.494775    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"]
Jul 24 00:14:23.494991 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.494914    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"
Jul 24 00:14:23.496989 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.496962    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 24 00:14:23.497238 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.497217    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 24 00:14:23.497361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.497262    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zg2vv\""
Jul 24 00:14:23.497432 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.497396    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 24 00:14:23.497542 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.497525    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 24 00:14:23.508650 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.508622    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"]
Jul 24 00:14:23.508750 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.508737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"
Jul 24 00:14:23.510844 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.510800    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 24 00:14:23.510962 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.510845    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 24 00:14:23.511022 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.510964    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 24 00:14:23.511022 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.510967    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 24 00:14:23.511130 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.511059    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-chg5p\""
Jul 24 00:14:23.530913 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.530883    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"]
Jul 24 00:14:23.530913 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.530916    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"]
Jul 24 00:14:23.531059 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.530935    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"]
Jul 24 00:14:23.531059 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.530985    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:23.532981 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.532959    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 24 00:14:23.533081 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.532997    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 24 00:14:23.533081 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.533065    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 24 00:14:23.533368 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.533352    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-krhsc\""
Jul 24 00:14:23.568971 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.568942    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"]
Jul 24 00:14:23.592796 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.592772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:23.592927 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.592804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7122daf-91a2-4e73-97d5-c1d236cb5a6b-serving-cert\") pod \"service-ca-operator-5d6c75f84f-4p6jk\" (UID: \"f7122daf-91a2-4e73-97d5-c1d236cb5a6b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"
Jul 24 00:14:23.592927 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.592853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwps\" (UniqueName: \"kubernetes.io/projected/f7122daf-91a2-4e73-97d5-c1d236cb5a6b-kube-api-access-hrwps\") pod \"service-ca-operator-5d6c75f84f-4p6jk\" (UID: \"f7122daf-91a2-4e73-97d5-c1d236cb5a6b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"
Jul 24 00:14:23.592927 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.592878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhplw\" (UniqueName: \"kubernetes.io/projected/5b5fb683-60bd-483c-a20b-51cd9a0285df-kube-api-access-xhplw\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt449\" (UID: \"5b5fb683-60bd-483c-a20b-51cd9a0285df\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"
Jul 24 00:14:23.592927 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.592898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b5fb683-60bd-483c-a20b-51cd9a0285df-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt449\" (UID: \"5b5fb683-60bd-483c-a20b-51cd9a0285df\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"
Jul 24 00:14:23.593198 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:23.592934    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 00:14:23.593198 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.592971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b5fb683-60bd-483c-a20b-51cd9a0285df-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt449\" (UID: \"5b5fb683-60bd-483c-a20b-51cd9a0285df\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"
Jul 24 00:14:23.593198 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:23.592998    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls podName:483be78f-e144-4a62-8932-88df4550a1af nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.592978006 +0000 UTC m=+52.160794137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls") pod "dns-default-kfm7w" (UID: "483be78f-e144-4a62-8932-88df4550a1af") : secret "dns-default-metrics-tls" not found
Jul 24 00:14:23.593198 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.593047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:23.593198 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.593076    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7122daf-91a2-4e73-97d5-c1d236cb5a6b-config\") pod \"service-ca-operator-5d6c75f84f-4p6jk\" (UID: \"f7122daf-91a2-4e73-97d5-c1d236cb5a6b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"
Jul 24 00:14:23.593198 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:23.593171    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 00:14:23.593198 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:23.593181    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-576d988ffd-2kxdz: secret "image-registry-tls" not found
Jul 24 00:14:23.593451 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:23.593223    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls podName:1fa5df10-e395-47e4-a3d8-bea5b049a19f nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.593209824 +0000 UTC m=+52.161025956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls") pod "image-registry-576d988ffd-2kxdz" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f") : secret "image-registry-tls" not found
Jul 24 00:14:23.594716 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.594698    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"]
Jul 24 00:14:23.594789 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.594778    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:23.596957 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.596939    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 00:14:23.597748 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.597729    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 24 00:14:23.597748 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.597739    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7hh5x\""
Jul 24 00:14:23.597905 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.597814    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 24 00:14:23.597970 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.597920    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 00:14:23.693702 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.693671    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g2th4\" (UID: \"5f02e80c-b8d3-439b-a5ab-7144d19a517f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:23.693702 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.693704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j22j8\" (UniqueName: \"kubernetes.io/projected/5f02e80c-b8d3-439b-a5ab-7144d19a517f-kube-api-access-j22j8\") pod \"cluster-samples-operator-6748489656-g2th4\" (UID: \"5f02e80c-b8d3-439b-a5ab-7144d19a517f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:23.693969 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.693727    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7122daf-91a2-4e73-97d5-c1d236cb5a6b-serving-cert\") pod \"service-ca-operator-5d6c75f84f-4p6jk\" (UID: \"f7122daf-91a2-4e73-97d5-c1d236cb5a6b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"
Jul 24 00:14:23.693969 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.693776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqbj\" (UniqueName: \"kubernetes.io/projected/cc5d1a31-2a44-4139-b308-bcf393724c12-kube-api-access-rhqbj\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:23.693969 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.693885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwps\" (UniqueName: \"kubernetes.io/projected/f7122daf-91a2-4e73-97d5-c1d236cb5a6b-kube-api-access-hrwps\") pod \"service-ca-operator-5d6c75f84f-4p6jk\" (UID: \"f7122daf-91a2-4e73-97d5-c1d236cb5a6b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"
Jul 24 00:14:23.693969 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.693912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc5d1a31-2a44-4139-b308-bcf393724c12-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:23.693969 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.693935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:23.694145 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.693974    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhplw\" (UniqueName: \"kubernetes.io/projected/5b5fb683-60bd-483c-a20b-51cd9a0285df-kube-api-access-xhplw\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt449\" (UID: \"5b5fb683-60bd-483c-a20b-51cd9a0285df\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"
Jul 24 00:14:23.694145 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.694005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b5fb683-60bd-483c-a20b-51cd9a0285df-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt449\" (UID: \"5b5fb683-60bd-483c-a20b-51cd9a0285df\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"
Jul 24 00:14:23.694145 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.694081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b5fb683-60bd-483c-a20b-51cd9a0285df-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt449\" (UID: \"5b5fb683-60bd-483c-a20b-51cd9a0285df\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"
Jul 24 00:14:23.694145 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.694132    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7122daf-91a2-4e73-97d5-c1d236cb5a6b-config\") pod \"service-ca-operator-5d6c75f84f-4p6jk\" (UID: \"f7122daf-91a2-4e73-97d5-c1d236cb5a6b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"
Jul 24 00:14:23.694756 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.694734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7122daf-91a2-4e73-97d5-c1d236cb5a6b-config\") pod \"service-ca-operator-5d6c75f84f-4p6jk\" (UID: \"f7122daf-91a2-4e73-97d5-c1d236cb5a6b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"
Jul 24 00:14:23.694879 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.694738    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b5fb683-60bd-483c-a20b-51cd9a0285df-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt449\" (UID: \"5b5fb683-60bd-483c-a20b-51cd9a0285df\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"
Jul 24 00:14:23.697290 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.697266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b5fb683-60bd-483c-a20b-51cd9a0285df-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt449\" (UID: \"5b5fb683-60bd-483c-a20b-51cd9a0285df\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"
Jul 24 00:14:23.697373 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.697266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7122daf-91a2-4e73-97d5-c1d236cb5a6b-serving-cert\") pod \"service-ca-operator-5d6c75f84f-4p6jk\" (UID: \"f7122daf-91a2-4e73-97d5-c1d236cb5a6b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"
Jul 24 00:14:23.705614 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.705584    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhplw\" (UniqueName: \"kubernetes.io/projected/5b5fb683-60bd-483c-a20b-51cd9a0285df-kube-api-access-xhplw\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-jt449\" (UID: \"5b5fb683-60bd-483c-a20b-51cd9a0285df\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"
Jul 24 00:14:23.705787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.705771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwps\" (UniqueName: \"kubernetes.io/projected/f7122daf-91a2-4e73-97d5-c1d236cb5a6b-kube-api-access-hrwps\") pod \"service-ca-operator-5d6c75f84f-4p6jk\" (UID: \"f7122daf-91a2-4e73-97d5-c1d236cb5a6b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"
Jul 24 00:14:23.795404 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.795310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g2th4\" (UID: \"5f02e80c-b8d3-439b-a5ab-7144d19a517f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:23.795404 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.795351    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j22j8\" (UniqueName: \"kubernetes.io/projected/5f02e80c-b8d3-439b-a5ab-7144d19a517f-kube-api-access-j22j8\") pod \"cluster-samples-operator-6748489656-g2th4\" (UID: \"5f02e80c-b8d3-439b-a5ab-7144d19a517f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:23.795404 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.795370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqbj\" (UniqueName: \"kubernetes.io/projected/cc5d1a31-2a44-4139-b308-bcf393724c12-kube-api-access-rhqbj\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:23.795404 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.795405    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc5d1a31-2a44-4139-b308-bcf393724c12-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:23.795718 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.795422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:23.795718 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:23.795452    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 00:14:23.795718 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:23.795533    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls podName:5f02e80c-b8d3-439b-a5ab-7144d19a517f nodeName:}" failed. No retries permitted until 2026-07-24 00:14:24.295514267 +0000 UTC m=+44.863330395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls") pod "cluster-samples-operator-6748489656-g2th4" (UID: "5f02e80c-b8d3-439b-a5ab-7144d19a517f") : secret "samples-operator-tls" not found
Jul 24 00:14:23.795718 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:23.795569    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:23.795718 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:23.795622    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls podName:cc5d1a31-2a44-4139-b308-bcf393724c12 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:24.295606678 +0000 UTC m=+44.863422817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8x9kb" (UID: "cc5d1a31-2a44-4139-b308-bcf393724c12") : secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:23.796160 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.796141    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc5d1a31-2a44-4139-b308-bcf393724c12-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:23.803617 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.803602    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"
Jul 24 00:14:23.806480 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.806453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j22j8\" (UniqueName: \"kubernetes.io/projected/5f02e80c-b8d3-439b-a5ab-7144d19a517f-kube-api-access-j22j8\") pod \"cluster-samples-operator-6748489656-g2th4\" (UID: \"5f02e80c-b8d3-439b-a5ab-7144d19a517f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:23.806578 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.806490    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqbj\" (UniqueName: \"kubernetes.io/projected/cc5d1a31-2a44-4139-b308-bcf393724c12-kube-api-access-rhqbj\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:23.816908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.816891    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"
Jul 24 00:14:23.957588 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:23.957562    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk"]
Jul 24 00:14:23.959324 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:23.959295    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7122daf_91a2_4e73_97d5_c1d236cb5a6b.slice/crio-ec2092a1b12058cae21adaa6dd6b2d5e4b150e065e166fe13c480b06170bcaf3 WatchSource:0}: Error finding container ec2092a1b12058cae21adaa6dd6b2d5e4b150e065e166fe13c480b06170bcaf3: Status 404 returned error can't find the container with id ec2092a1b12058cae21adaa6dd6b2d5e4b150e065e166fe13c480b06170bcaf3
Jul 24 00:14:24.006092 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:24.006060    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449"]
Jul 24 00:14:24.009376 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:24.009350    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5fb683_60bd_483c_a20b_51cd9a0285df.slice/crio-cd830bb29ff7eedf76ab8afefbfcdbb457ac9f53e8d8ec8c3e44adaeb4f593fe WatchSource:0}: Error finding container cd830bb29ff7eedf76ab8afefbfcdbb457ac9f53e8d8ec8c3e44adaeb4f593fe: Status 404 returned error can't find the container with id cd830bb29ff7eedf76ab8afefbfcdbb457ac9f53e8d8ec8c3e44adaeb4f593fe
Jul 24 00:14:24.166334 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:24.166232    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449" event={"ID":"5b5fb683-60bd-483c-a20b-51cd9a0285df","Type":"ContainerStarted","Data":"cd830bb29ff7eedf76ab8afefbfcdbb457ac9f53e8d8ec8c3e44adaeb4f593fe"}
Jul 24 00:14:24.167330 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:24.167302    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk" event={"ID":"f7122daf-91a2-4e73-97d5-c1d236cb5a6b","Type":"ContainerStarted","Data":"ec2092a1b12058cae21adaa6dd6b2d5e4b150e065e166fe13c480b06170bcaf3"}
Jul 24 00:14:24.300758 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:24.300718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g2th4\" (UID: \"5f02e80c-b8d3-439b-a5ab-7144d19a517f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:24.300979 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:24.300786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:24.300979 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:24.300905    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 00:14:24.300979 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:24.300941    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:24.301145 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:24.300986    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls podName:5f02e80c-b8d3-439b-a5ab-7144d19a517f nodeName:}" failed. No retries permitted until 2026-07-24 00:14:25.300966158 +0000 UTC m=+45.868782300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls") pod "cluster-samples-operator-6748489656-g2th4" (UID: "5f02e80c-b8d3-439b-a5ab-7144d19a517f") : secret "samples-operator-tls" not found
Jul 24 00:14:24.301145 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:24.301006    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls podName:cc5d1a31-2a44-4139-b308-bcf393724c12 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:25.300997519 +0000 UTC m=+45.868813649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8x9kb" (UID: "cc5d1a31-2a44-4139-b308-bcf393724c12") : secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:25.307138 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:25.307099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g2th4\" (UID: \"5f02e80c-b8d3-439b-a5ab-7144d19a517f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:25.307541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:25.307155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:25.307541 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:25.307259    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 00:14:25.307541 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:25.307266    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:25.307541 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:25.307333    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls podName:5f02e80c-b8d3-439b-a5ab-7144d19a517f nodeName:}" failed. No retries permitted until 2026-07-24 00:14:27.307313672 +0000 UTC m=+47.875129808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls") pod "cluster-samples-operator-6748489656-g2th4" (UID: "5f02e80c-b8d3-439b-a5ab-7144d19a517f") : secret "samples-operator-tls" not found
Jul 24 00:14:25.307541 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:25.307355    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls podName:cc5d1a31-2a44-4139-b308-bcf393724c12 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:27.307343818 +0000 UTC m=+47.875159955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8x9kb" (UID: "cc5d1a31-2a44-4139-b308-bcf393724c12") : secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:27.326203 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:27.326167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:27.326519 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:27.326314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g2th4\" (UID: \"5f02e80c-b8d3-439b-a5ab-7144d19a517f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:27.326519 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:27.326476    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 00:14:27.326631 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:27.326536    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls podName:5f02e80c-b8d3-439b-a5ab-7144d19a517f nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.326517958 +0000 UTC m=+51.894334106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls") pod "cluster-samples-operator-6748489656-g2th4" (UID: "5f02e80c-b8d3-439b-a5ab-7144d19a517f") : secret "samples-operator-tls" not found
Jul 24 00:14:27.326631 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:27.326575    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:27.326631 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:27.326613    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls podName:cc5d1a31-2a44-4139-b308-bcf393724c12 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.326602358 +0000 UTC m=+51.894418489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8x9kb" (UID: "cc5d1a31-2a44-4139-b308-bcf393724c12") : secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:28.178345 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:28.178306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449" event={"ID":"5b5fb683-60bd-483c-a20b-51cd9a0285df","Type":"ContainerStarted","Data":"9cfda213352dee5f86b6483968d88cda4a81a7e07ffb21c0b30e9939d49ab9b9"}
Jul 24 00:14:28.179793 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:28.179765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk" event={"ID":"f7122daf-91a2-4e73-97d5-c1d236cb5a6b","Type":"ContainerStarted","Data":"204551c49bd70e6b533a2ccd35f04b6ce6e3aa577d88a563dda7afa8ffb806de"}
Jul 24 00:14:28.196105 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:28.196053    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449" podStartSLOduration=2.00827497 podStartE2EDuration="5.196039275s" totalImagesPullingTime="3.187764305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:28.194760448 +0000 UTC m=+48.762576598" watchObservedRunningTime="2026-07-24 00:14:28.196039275 +0000 UTC m=+48.763855426"
Jul 24 00:14:28.208228 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:28.208168    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk" podStartSLOduration=1.965602276 podStartE2EDuration="5.208145292s" totalImagesPullingTime="3.242543016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:28.207734021 +0000 UTC m=+48.775550184" watchObservedRunningTime="2026-07-24 00:14:28.208145292 +0000 UTC m=+48.775961443"
Jul 24 00:14:28.903071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:28.903034    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xhght"]
Jul 24 00:14:28.924616 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:28.924581    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xhght"]
Jul 24 00:14:28.924778 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:28.924710    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xhght"
Jul 24 00:14:28.926974 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:28.926947    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 24 00:14:28.926974 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:28.926960    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kjnt7\""
Jul 24 00:14:28.927800 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:28.927785    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 24 00:14:29.044393 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.044355    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msj5f\" (UniqueName: \"kubernetes.io/projected/4e393c13-3ca6-4f6c-9687-8447041dd5f5-kube-api-access-msj5f\") pod \"migrator-6865c9967-xhght\" (UID: \"4e393c13-3ca6-4f6c-9687-8447041dd5f5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xhght"
Jul 24 00:14:29.145750 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.145710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msj5f\" (UniqueName: \"kubernetes.io/projected/4e393c13-3ca6-4f6c-9687-8447041dd5f5-kube-api-access-msj5f\") pod \"migrator-6865c9967-xhght\" (UID: \"4e393c13-3ca6-4f6c-9687-8447041dd5f5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xhght"
Jul 24 00:14:29.153655 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.153591    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msj5f\" (UniqueName: \"kubernetes.io/projected/4e393c13-3ca6-4f6c-9687-8447041dd5f5-kube-api-access-msj5f\") pod \"migrator-6865c9967-xhght\" (UID: \"4e393c13-3ca6-4f6c-9687-8447041dd5f5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xhght"
Jul 24 00:14:29.234347 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.234314    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xhght"
Jul 24 00:14:29.374876 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.374824    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-xhght"]
Jul 24 00:14:29.389142 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:29.389113    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e393c13_3ca6_4f6c_9687_8447041dd5f5.slice/crio-cc96b72b65c1c8366728859287a287fcc50ef5fdc1e79c4f88976769c1067eb5 WatchSource:0}: Error finding container cc96b72b65c1c8366728859287a287fcc50ef5fdc1e79c4f88976769c1067eb5: Status 404 returned error can't find the container with id cc96b72b65c1c8366728859287a287fcc50ef5fdc1e79c4f88976769c1067eb5
Jul 24 00:14:29.573424 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.573389    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w86dg"]
Jul 24 00:14:29.576029 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.576011    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.578417 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.578391    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 00:14:29.579184 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.579161    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 00:14:29.579269 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.579206    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 00:14:29.579269 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.579226    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 00:14:29.579269 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.579206    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mtjsg\""
Jul 24 00:14:29.587048 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.587022    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w86dg"]
Jul 24 00:14:29.752331 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.752295    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9da248de-af65-442a-9766-eae338141049-crio-socket\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.752495 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.752361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.752495 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.752423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9da248de-af65-442a-9766-eae338141049-data-volume\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.752495 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.752450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9da248de-af65-442a-9766-eae338141049-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.752495 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.752471    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8t5j\" (UniqueName: \"kubernetes.io/projected/9da248de-af65-442a-9766-eae338141049-kube-api-access-b8t5j\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.853365 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.853280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.853511 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:29.853397    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 00:14:29.853511 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.853423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9da248de-af65-442a-9766-eae338141049-data-volume\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.853511 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.853451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9da248de-af65-442a-9766-eae338141049-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.853511 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:29.853463    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls podName:9da248de-af65-442a-9766-eae338141049 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:30.35344805 +0000 UTC m=+50.921264178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w86dg" (UID: "9da248de-af65-442a-9766-eae338141049") : secret "insights-runtime-extractor-tls" not found
Jul 24 00:14:29.853511 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.853491    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8t5j\" (UniqueName: \"kubernetes.io/projected/9da248de-af65-442a-9766-eae338141049-kube-api-access-b8t5j\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.853511 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:29.853508    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 00:14:29.853697 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:29.853571    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9da248de-af65-442a-9766-eae338141049-kube-rbac-proxy-cm podName:9da248de-af65-442a-9766-eae338141049 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:30.353550626 +0000 UTC m=+50.921366763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9da248de-af65-442a-9766-eae338141049-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w86dg" (UID: "9da248de-af65-442a-9766-eae338141049") : configmap "kube-rbac-proxy" not found
Jul 24 00:14:29.853739 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.853696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9da248de-af65-442a-9766-eae338141049-crio-socket\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.853770 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.853746    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9da248de-af65-442a-9766-eae338141049-data-volume\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.853916 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.853901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9da248de-af65-442a-9766-eae338141049-crio-socket\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:29.865011 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:29.864990    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8t5j\" (UniqueName: \"kubernetes.io/projected/9da248de-af65-442a-9766-eae338141049-kube-api-access-b8t5j\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:30.187332 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.187242    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xhght" event={"ID":"4e393c13-3ca6-4f6c-9687-8447041dd5f5","Type":"ContainerStarted","Data":"cc96b72b65c1c8366728859287a287fcc50ef5fdc1e79c4f88976769c1067eb5"}
Jul 24 00:14:30.357572 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.357534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:30.357754 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.357600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9da248de-af65-442a-9766-eae338141049-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:30.357754 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:30.357676    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 00:14:30.357754 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:30.357726    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 00:14:30.357754 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:30.357737    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls podName:9da248de-af65-442a-9766-eae338141049 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.357722214 +0000 UTC m=+51.925538342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w86dg" (UID: "9da248de-af65-442a-9766-eae338141049") : secret "insights-runtime-extractor-tls" not found
Jul 24 00:14:30.357972 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:30.357776    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9da248de-af65-442a-9766-eae338141049-kube-rbac-proxy-cm podName:9da248de-af65-442a-9766-eae338141049 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:31.357762325 +0000 UTC m=+51.925578452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9da248de-af65-442a-9766-eae338141049-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w86dg" (UID: "9da248de-af65-442a-9766-eae338141049") : configmap "kube-rbac-proxy" not found
Jul 24 00:14:30.561420 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.561385    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-tvlzv"]
Jul 24 00:14:30.563502 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.563473    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv"
Jul 24 00:14:30.565978 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.565954    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 24 00:14:30.566124 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.565991    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 24 00:14:30.566847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.566766    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 24 00:14:30.566847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.566795    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-s9gt8\""
Jul 24 00:14:30.566847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.566815    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 24 00:14:30.574049 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.574027    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-tvlzv"]
Jul 24 00:14:30.661160 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.661135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc39a078-162f-4c3e-bcac-91312b2f1cb1-signing-cabundle\") pod \"service-ca-658d7b7c78-tvlzv\" (UID: \"dc39a078-162f-4c3e-bcac-91312b2f1cb1\") " pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv"
Jul 24 00:14:30.661252 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.661192    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc39a078-162f-4c3e-bcac-91312b2f1cb1-signing-key\") pod \"service-ca-658d7b7c78-tvlzv\" (UID: \"dc39a078-162f-4c3e-bcac-91312b2f1cb1\") " pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv"
Jul 24 00:14:30.661293 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.661279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55xlm\" (UniqueName: \"kubernetes.io/projected/dc39a078-162f-4c3e-bcac-91312b2f1cb1-kube-api-access-55xlm\") pod \"service-ca-658d7b7c78-tvlzv\" (UID: \"dc39a078-162f-4c3e-bcac-91312b2f1cb1\") " pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv"
Jul 24 00:14:30.762146 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.762116    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc39a078-162f-4c3e-bcac-91312b2f1cb1-signing-key\") pod \"service-ca-658d7b7c78-tvlzv\" (UID: \"dc39a078-162f-4c3e-bcac-91312b2f1cb1\") " pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv"
Jul 24 00:14:30.762309 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.762211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55xlm\" (UniqueName: \"kubernetes.io/projected/dc39a078-162f-4c3e-bcac-91312b2f1cb1-kube-api-access-55xlm\") pod \"service-ca-658d7b7c78-tvlzv\" (UID: \"dc39a078-162f-4c3e-bcac-91312b2f1cb1\") " pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv"
Jul 24 00:14:30.762309 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.762284    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc39a078-162f-4c3e-bcac-91312b2f1cb1-signing-cabundle\") pod \"service-ca-658d7b7c78-tvlzv\" (UID: \"dc39a078-162f-4c3e-bcac-91312b2f1cb1\") " pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv"
Jul 24 00:14:30.763041 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.763016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc39a078-162f-4c3e-bcac-91312b2f1cb1-signing-cabundle\") pod \"service-ca-658d7b7c78-tvlzv\" (UID: \"dc39a078-162f-4c3e-bcac-91312b2f1cb1\") " pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv"
Jul 24 00:14:30.764429 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.764406    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc39a078-162f-4c3e-bcac-91312b2f1cb1-signing-key\") pod \"service-ca-658d7b7c78-tvlzv\" (UID: \"dc39a078-162f-4c3e-bcac-91312b2f1cb1\") " pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv"
Jul 24 00:14:30.771534 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.771514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55xlm\" (UniqueName: \"kubernetes.io/projected/dc39a078-162f-4c3e-bcac-91312b2f1cb1-kube-api-access-55xlm\") pod \"service-ca-658d7b7c78-tvlzv\" (UID: \"dc39a078-162f-4c3e-bcac-91312b2f1cb1\") " pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv"
Jul 24 00:14:30.873806 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.873710    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv"
Jul 24 00:14:30.991284 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:30.991245    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-tvlzv"]
Jul 24 00:14:30.994191 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:30.994166    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc39a078_162f_4c3e_bcac_91312b2f1cb1.slice/crio-d575911ae44d4d357f2f29be4a8dd1c5864051cac336e6d82588d8073263c499 WatchSource:0}: Error finding container d575911ae44d4d357f2f29be4a8dd1c5864051cac336e6d82588d8073263c499: Status 404 returned error can't find the container with id d575911ae44d4d357f2f29be4a8dd1c5864051cac336e6d82588d8073263c499
Jul 24 00:14:31.192544 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.192442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv" event={"ID":"dc39a078-162f-4c3e-bcac-91312b2f1cb1","Type":"ContainerStarted","Data":"db75dff4c957225e5abd95a5c4685f369f2e6eb53404760c11e16e3439f78abe"}
Jul 24 00:14:31.192544 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.192504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv" event={"ID":"dc39a078-162f-4c3e-bcac-91312b2f1cb1","Type":"ContainerStarted","Data":"d575911ae44d4d357f2f29be4a8dd1c5864051cac336e6d82588d8073263c499"}
Jul 24 00:14:31.194216 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.194146    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xhght" event={"ID":"4e393c13-3ca6-4f6c-9687-8447041dd5f5","Type":"ContainerStarted","Data":"f9bd6c93404f12f39115001c7ea91b37d4660b99af0455e5c19017478e45ec02"}
Jul 24 00:14:31.194333 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.194242    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xhght" event={"ID":"4e393c13-3ca6-4f6c-9687-8447041dd5f5","Type":"ContainerStarted","Data":"8592dc2c1948d1ecf5875f5c287037f79e9bac3c0210f14646002c31a9a8f1f4"}
Jul 24 00:14:31.224414 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.224370    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-tvlzv" podStartSLOduration=1.22435471 podStartE2EDuration="1.22435471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:31.223682345 +0000 UTC m=+51.791498495" watchObservedRunningTime="2026-07-24 00:14:31.22435471 +0000 UTC m=+51.792170841"
Jul 24 00:14:31.274608 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.274566    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-xhght" podStartSLOduration=2.026402556 podStartE2EDuration="3.27455158s" totalImagesPullingTime="1.248149024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:31.249425536 +0000 UTC m=+51.817241686" watchObservedRunningTime="2026-07-24 00:14:31.27455158 +0000 UTC m=+51.842367731"
Jul 24 00:14:31.367651 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.367615    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g2th4\" (UID: \"5f02e80c-b8d3-439b-a5ab-7144d19a517f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:31.367806 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.367687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:31.367806 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.367727    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9da248de-af65-442a-9766-eae338141049-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:31.367806 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:31.367729    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 00:14:31.367806 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:31.367788    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls podName:5f02e80c-b8d3-439b-a5ab-7144d19a517f nodeName:}" failed. No retries permitted until 2026-07-24 00:14:39.367773961 +0000 UTC m=+59.935590092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls") pod "cluster-samples-operator-6748489656-g2th4" (UID: "5f02e80c-b8d3-439b-a5ab-7144d19a517f") : secret "samples-operator-tls" not found
Jul 24 00:14:31.367996 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.367816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:31.367996 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:31.367852    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 00:14:31.367996 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:31.367911    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls podName:9da248de-af65-442a-9766-eae338141049 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:33.367894665 +0000 UTC m=+53.935710811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w86dg" (UID: "9da248de-af65-442a-9766-eae338141049") : secret "insights-runtime-extractor-tls" not found
Jul 24 00:14:31.367996 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:31.367922    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:31.367996 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:31.367974    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls podName:cc5d1a31-2a44-4139-b308-bcf393724c12 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:39.367958556 +0000 UTC m=+59.935774692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8x9kb" (UID: "cc5d1a31-2a44-4139-b308-bcf393724c12") : secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:31.368213 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.368198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9da248de-af65-442a-9766-eae338141049-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:31.670617 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.670578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:31.670774 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:31.670635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:31.670774 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:31.670744    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 00:14:31.670774 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:31.670765    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-576d988ffd-2kxdz: secret "image-registry-tls" not found
Jul 24 00:14:31.670920 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:31.670825    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls podName:1fa5df10-e395-47e4-a3d8-bea5b049a19f nodeName:}" failed. No retries permitted until 2026-07-24 00:14:47.67080786 +0000 UTC m=+68.238624006 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls") pod "image-registry-576d988ffd-2kxdz" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f") : secret "image-registry-tls" not found
Jul 24 00:14:31.670920 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:31.670892    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 00:14:31.670999 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:31.670942    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls podName:483be78f-e144-4a62-8932-88df4550a1af nodeName:}" failed. No retries permitted until 2026-07-24 00:14:47.670928876 +0000 UTC m=+68.238745009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls") pod "dns-default-kfm7w" (UID: "483be78f-e144-4a62-8932-88df4550a1af") : secret "dns-default-metrics-tls" not found
Jul 24 00:14:32.176127 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:32.176102    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mttpm_a00cf2c5-5c20-410f-bae2-80170581b317/dns-node-resolver/0.log"
Jul 24 00:14:32.975671 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:32.975637    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kd9fr_6d50d211-9d46-4e14-ace4-554592a1ddc0/node-ca/0.log"
Jul 24 00:14:33.386307 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:33.386208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:33.386452 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:33.386361    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 00:14:33.386452 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:33.386448    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls podName:9da248de-af65-442a-9766-eae338141049 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:37.386411163 +0000 UTC m=+57.954227294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w86dg" (UID: "9da248de-af65-442a-9766-eae338141049") : secret "insights-runtime-extractor-tls" not found
Jul 24 00:14:33.776035 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:33.775977    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7rbjj_ac29741e-a8ad-4a01-8342-e631c2eca6ce/serve-healthcheck-canary/0.log"
Jul 24 00:14:33.975522 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:33.975494    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-xhght_4e393c13-3ca6-4f6c-9687-8447041dd5f5/migrator/0.log"
Jul 24 00:14:34.175972 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:34.175895    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-xhght_4e393c13-3ca6-4f6c-9687-8447041dd5f5/graceful-termination/0.log"
Jul 24 00:14:34.376667 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:34.376621    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-jt449_5b5fb683-60bd-483c-a20b-51cd9a0285df/kube-storage-version-migrator-operator/0.log"
Jul 24 00:14:37.419472 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:37.419433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:37.421653 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:37.421631    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9da248de-af65-442a-9766-eae338141049-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w86dg\" (UID: \"9da248de-af65-442a-9766-eae338141049\") " pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:37.685095 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:37.685000    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w86dg"
Jul 24 00:14:37.809926 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:37.809859    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w86dg"]
Jul 24 00:14:37.812401 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:37.812369    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da248de_af65_442a_9766_eae338141049.slice/crio-861d9e17a8a7e1946b54c8e00b8549986b74c91c4d68bef40fe9ab70897dfe28 WatchSource:0}: Error finding container 861d9e17a8a7e1946b54c8e00b8549986b74c91c4d68bef40fe9ab70897dfe28: Status 404 returned error can't find the container with id 861d9e17a8a7e1946b54c8e00b8549986b74c91c4d68bef40fe9ab70897dfe28
Jul 24 00:14:38.216530 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:38.216496    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w86dg" event={"ID":"9da248de-af65-442a-9766-eae338141049","Type":"ContainerStarted","Data":"861d9e17a8a7e1946b54c8e00b8549986b74c91c4d68bef40fe9ab70897dfe28"}
Jul 24 00:14:39.125534 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:39.125500    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-62qhx"
Jul 24 00:14:39.439279 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:39.439194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:39.439455 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:39.439302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g2th4\" (UID: \"5f02e80c-b8d3-439b-a5ab-7144d19a517f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:39.439455 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:39.439342    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:39.439455 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:14:39.439414    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls podName:cc5d1a31-2a44-4139-b308-bcf393724c12 nodeName:}" failed. No retries permitted until 2026-07-24 00:14:55.439392548 +0000 UTC m=+76.007208693 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8x9kb" (UID: "cc5d1a31-2a44-4139-b308-bcf393724c12") : secret "cluster-monitoring-operator-tls" not found
Jul 24 00:14:39.442327 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:39.442275    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f02e80c-b8d3-439b-a5ab-7144d19a517f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g2th4\" (UID: \"5f02e80c-b8d3-439b-a5ab-7144d19a517f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:39.739784 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:39.739739    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"
Jul 24 00:14:39.899484 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:39.899292    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4"]
Jul 24 00:14:40.224422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:40.224385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4" event={"ID":"5f02e80c-b8d3-439b-a5ab-7144d19a517f","Type":"ContainerStarted","Data":"65b0dcec8298ed7c4c6848c380b53ea1d4283c9011779ae024b44d3c920f21b9"}
Jul 24 00:14:43.234974 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:43.234936    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w86dg" event={"ID":"9da248de-af65-442a-9766-eae338141049","Type":"ContainerStarted","Data":"90923c54a4e6a3cc45fa2fa3e24b2289563b90f48a5e5556cf3df88ed9311d5a"}
Jul 24 00:14:44.240337 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:44.240298    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4" event={"ID":"5f02e80c-b8d3-439b-a5ab-7144d19a517f","Type":"ContainerStarted","Data":"7b26c05f8da1d8e271af0473aae5dd5346191271fbdea891496bcfdda998b6a8"}
Jul 24 00:14:44.240801 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:44.240368    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4" event={"ID":"5f02e80c-b8d3-439b-a5ab-7144d19a517f","Type":"ContainerStarted","Data":"a9f81991f5b8b1c1bc496c25dcda68b2e19604444794cca8038e802912e18e2b"}
Jul 24 00:14:44.256052 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:44.255997    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g2th4" podStartSLOduration=17.584132012 podStartE2EDuration="21.255979119s" totalImagesPullingTime="3.671847107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:44.254403887 +0000 UTC m=+64.822220051" watchObservedRunningTime="2026-07-24 00:14:44.255979119 +0000 UTC m=+64.823795268"
Jul 24 00:14:45.801540 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:45.801496    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:45.801992 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:45.801569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:45.804166 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:45.804135    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 00:14:45.804924 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:45.804903    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 00:14:45.815410 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:45.815382    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f46a43-3c16-45f8-90c4-0f815e6850d0-original-pull-secret\") pod \"global-pull-secret-syncer-kgsn8\" (UID: \"71f46a43-3c16-45f8-90c4-0f815e6850d0\") " pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:45.815546 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:45.815511    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b63b995d-5404-4953-acf6-31e068d8cd35-metrics-certs\") pod \"network-metrics-daemon-7kps9\" (UID: \"b63b995d-5404-4953-acf6-31e068d8cd35\") " pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:45.902984 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:45.902935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xb6\" (UniqueName: \"kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6\") pod \"network-check-target-r722z\" (UID: \"1bae97a1-6b3a-4ff1-847b-2df09e067715\") " pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:45.905560 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:45.905536    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 00:14:45.915905 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:45.915884    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 00:14:45.927394 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:45.927362    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2xb6\" (UniqueName: \"kubernetes.io/projected/1bae97a1-6b3a-4ff1-847b-2df09e067715-kube-api-access-r2xb6\") pod \"network-check-target-r722z\" (UID: \"1bae97a1-6b3a-4ff1-847b-2df09e067715\") " pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:46.077535 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:46.077455    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gpdt7\""
Jul 24 00:14:46.082454 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:46.082426    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kgsn8"
Jul 24 00:14:46.085577 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:46.085550    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:46.090340 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:46.090308    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jq7gh\""
Jul 24 00:14:46.098287 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:46.098265    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7kps9"
Jul 24 00:14:46.755623 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:46.755594    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r722z"]
Jul 24 00:14:46.758337 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:46.758302    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bae97a1_6b3a_4ff1_847b_2df09e067715.slice/crio-6fd57c6248bc9326338e113bd5abaecb661eb78e59ee68b406cf98262e385903 WatchSource:0}: Error finding container 6fd57c6248bc9326338e113bd5abaecb661eb78e59ee68b406cf98262e385903: Status 404 returned error can't find the container with id 6fd57c6248bc9326338e113bd5abaecb661eb78e59ee68b406cf98262e385903
Jul 24 00:14:46.976066 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:46.976030    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kgsn8"]
Jul 24 00:14:46.979499 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:46.979462    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7kps9"]
Jul 24 00:14:46.979741 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:46.979677    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f46a43_3c16_45f8_90c4_0f815e6850d0.slice/crio-b1b754ea126902aa7493f1ba3163743fb0e90ca36e30f40a9bb64dc3094399f6 WatchSource:0}: Error finding container b1b754ea126902aa7493f1ba3163743fb0e90ca36e30f40a9bb64dc3094399f6: Status 404 returned error can't find the container with id b1b754ea126902aa7493f1ba3163743fb0e90ca36e30f40a9bb64dc3094399f6
Jul 24 00:14:46.981698 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:46.981676    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63b995d_5404_4953_acf6_31e068d8cd35.slice/crio-4cd1077e6deb1196af8ef80b7376a6950aa22ce37c6e20d510e6e458c37b652a WatchSource:0}: Error finding container 4cd1077e6deb1196af8ef80b7376a6950aa22ce37c6e20d510e6e458c37b652a: Status 404 returned error can't find the container with id 4cd1077e6deb1196af8ef80b7376a6950aa22ce37c6e20d510e6e458c37b652a
Jul 24 00:14:47.252460 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.252373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w86dg" event={"ID":"9da248de-af65-442a-9766-eae338141049","Type":"ContainerStarted","Data":"2e6b16aed8b9c1ae6cea700e62c968fa6a8164858a7b73dd7be6ae08780a7db5"}
Jul 24 00:14:47.253907 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.253871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7kps9" event={"ID":"b63b995d-5404-4953-acf6-31e068d8cd35","Type":"ContainerStarted","Data":"4cd1077e6deb1196af8ef80b7376a6950aa22ce37c6e20d510e6e458c37b652a"}
Jul 24 00:14:47.255135 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.255081    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kgsn8" event={"ID":"71f46a43-3c16-45f8-90c4-0f815e6850d0","Type":"ContainerStarted","Data":"b1b754ea126902aa7493f1ba3163743fb0e90ca36e30f40a9bb64dc3094399f6"}
Jul 24 00:14:47.256413 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.256391    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r722z" event={"ID":"1bae97a1-6b3a-4ff1-847b-2df09e067715","Type":"ContainerStarted","Data":"6fd57c6248bc9326338e113bd5abaecb661eb78e59ee68b406cf98262e385903"}
Jul 24 00:14:47.719575 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.719535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:47.719774 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.719609    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:47.723497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.723428    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls\") pod \"image-registry-576d988ffd-2kxdz\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") " pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:47.724076 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.724026    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483be78f-e144-4a62-8932-88df4550a1af-metrics-tls\") pod \"dns-default-kfm7w\" (UID: \"483be78f-e144-4a62-8932-88df4550a1af\") " pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:47.889528 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.889496    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qlxsm\""
Jul 24 00:14:47.897521 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.897485    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:47.902082 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.902056    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvjcq\""
Jul 24 00:14:47.910023 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:47.909992    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:48.422421 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:48.422316    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kfm7w"]
Jul 24 00:14:48.448385 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:48.448355    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-576d988ffd-2kxdz"]
Jul 24 00:14:48.593603 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:48.593433    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483be78f_e144_4a62_8932_88df4550a1af.slice/crio-a0c0c0b491a3ab8e0508e38a81090b02d8379002811bbddfac77719b8f3634a8 WatchSource:0}: Error finding container a0c0c0b491a3ab8e0508e38a81090b02d8379002811bbddfac77719b8f3634a8: Status 404 returned error can't find the container with id a0c0c0b491a3ab8e0508e38a81090b02d8379002811bbddfac77719b8f3634a8
Jul 24 00:14:48.596795 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:48.596763    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa5df10_e395_47e4_a3d8_bea5b049a19f.slice/crio-49aa1fd7985a4d0ce3b20cf6e0c8e2dafcd895775bb32e4974efaf02b22da4c5 WatchSource:0}: Error finding container 49aa1fd7985a4d0ce3b20cf6e0c8e2dafcd895775bb32e4974efaf02b22da4c5: Status 404 returned error can't find the container with id 49aa1fd7985a4d0ce3b20cf6e0c8e2dafcd895775bb32e4974efaf02b22da4c5
Jul 24 00:14:49.268361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:49.268321    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7kps9" event={"ID":"b63b995d-5404-4953-acf6-31e068d8cd35","Type":"ContainerStarted","Data":"fb6f21aa5e09ffd12505dfe8948926894b74fb8303f8bf9c41f1195a9556f316"}
Jul 24 00:14:49.268546 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:49.268393    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7kps9" event={"ID":"b63b995d-5404-4953-acf6-31e068d8cd35","Type":"ContainerStarted","Data":"658c34a5379e3cf50325c2b8ad12d198a0bc38e02d6e3b9da7d8fe91bd1c77ce"}
Jul 24 00:14:49.271588 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:49.271556    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz" event={"ID":"1fa5df10-e395-47e4-a3d8-bea5b049a19f","Type":"ContainerStarted","Data":"7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed"}
Jul 24 00:14:49.271745 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:49.271608    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz" event={"ID":"1fa5df10-e395-47e4-a3d8-bea5b049a19f","Type":"ContainerStarted","Data":"49aa1fd7985a4d0ce3b20cf6e0c8e2dafcd895775bb32e4974efaf02b22da4c5"}
Jul 24 00:14:49.272115 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:49.272094    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:14:49.274028 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:49.274001    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kfm7w" event={"ID":"483be78f-e144-4a62-8932-88df4550a1af","Type":"ContainerStarted","Data":"a0c0c0b491a3ab8e0508e38a81090b02d8379002811bbddfac77719b8f3634a8"}
Jul 24 00:14:49.286055 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:49.284742    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7kps9" podStartSLOduration=67.641784631 podStartE2EDuration="1m9.284722777s" totalImagesPullingTime="1.642938146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:49.284223794 +0000 UTC m=+69.852039950" watchObservedRunningTime="2026-07-24 00:14:49.284722777 +0000 UTC m=+69.852538927"
Jul 24 00:14:49.302005 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:49.301954    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz" podStartSLOduration=68.301936739 podStartE2EDuration="1m8.301936739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:49.30086331 +0000 UTC m=+69.868679496" watchObservedRunningTime="2026-07-24 00:14:49.301936739 +0000 UTC m=+69.869752891"
Jul 24 00:14:53.406034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.405979    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s"]
Jul 24 00:14:53.408344 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.408316    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"]
Jul 24 00:14:53.408484 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.408457    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s"
Jul 24 00:14:53.410283 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.410261    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.411179 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.411152    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 00:14:53.411285 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.411158    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8fllh\""
Jul 24 00:14:53.411500 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.411478    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 00:14:53.411619 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.411506    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 00:14:53.412703 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.412678    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 24 00:14:53.413032 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.412881    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 24 00:14:53.413032 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.412955    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 24 00:14:53.413032 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.412990    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 24 00:14:53.414026 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.414005    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 24 00:14:53.425772 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.425703    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s"]
Jul 24 00:14:53.426862 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.426816    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"]
Jul 24 00:14:53.578614 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.578567    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.578614 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.578611    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.578902 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.578647    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhnnw\" (UniqueName: \"kubernetes.io/projected/962628cc-0a5b-43fc-91a1-6300cc786c6e-kube-api-access-mhnnw\") pod \"managed-serviceaccount-addon-agent-f5669ffdd-wlm8s\" (UID: \"962628cc-0a5b-43fc-91a1-6300cc786c6e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s"
Jul 24 00:14:53.578902 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.578671    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/355598f1-a660-4e77-bca5-1b719fd5bc72-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.578902 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.578781    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-hub\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.578902 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.578823    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/962628cc-0a5b-43fc-91a1-6300cc786c6e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f5669ffdd-wlm8s\" (UID: \"962628cc-0a5b-43fc-91a1-6300cc786c6e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s"
Jul 24 00:14:53.578902 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.578869    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-ca\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.579116 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.578906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwlh8\" (UniqueName: \"kubernetes.io/projected/355598f1-a660-4e77-bca5-1b719fd5bc72-kube-api-access-dwlh8\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.679954 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.679859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.679954 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.679907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.679954 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.679946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhnnw\" (UniqueName: \"kubernetes.io/projected/962628cc-0a5b-43fc-91a1-6300cc786c6e-kube-api-access-mhnnw\") pod \"managed-serviceaccount-addon-agent-f5669ffdd-wlm8s\" (UID: \"962628cc-0a5b-43fc-91a1-6300cc786c6e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s"
Jul 24 00:14:53.680232 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.679973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/355598f1-a660-4e77-bca5-1b719fd5bc72-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.680232 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.680027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-hub\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.680232 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.680056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/962628cc-0a5b-43fc-91a1-6300cc786c6e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f5669ffdd-wlm8s\" (UID: \"962628cc-0a5b-43fc-91a1-6300cc786c6e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s"
Jul 24 00:14:53.680232 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.680081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-ca\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.680232 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.680115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwlh8\" (UniqueName: \"kubernetes.io/projected/355598f1-a660-4e77-bca5-1b719fd5bc72-kube-api-access-dwlh8\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.680846 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.680790    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/355598f1-a660-4e77-bca5-1b719fd5bc72-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.683178 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.683146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.683320 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.683231    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.683382 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.683351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-hub\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.683894 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.683870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/355598f1-a660-4e77-bca5-1b719fd5bc72-ca\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.684002 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.683924    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/962628cc-0a5b-43fc-91a1-6300cc786c6e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f5669ffdd-wlm8s\" (UID: \"962628cc-0a5b-43fc-91a1-6300cc786c6e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s"
Jul 24 00:14:53.688546 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.688519    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwlh8\" (UniqueName: \"kubernetes.io/projected/355598f1-a660-4e77-bca5-1b719fd5bc72-kube-api-access-dwlh8\") pod \"cluster-proxy-proxy-agent-6695d6c4b5-wrh5b\" (UID: \"355598f1-a660-4e77-bca5-1b719fd5bc72\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:53.688753 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.688731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhnnw\" (UniqueName: \"kubernetes.io/projected/962628cc-0a5b-43fc-91a1-6300cc786c6e-kube-api-access-mhnnw\") pod \"managed-serviceaccount-addon-agent-f5669ffdd-wlm8s\" (UID: \"962628cc-0a5b-43fc-91a1-6300cc786c6e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s"
Jul 24 00:14:53.728527 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.728487    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s"
Jul 24 00:14:53.734236 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:53.734205    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"
Jul 24 00:14:54.203488 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.203455    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s"]
Jul 24 00:14:54.207279 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:54.207066    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod962628cc_0a5b_43fc_91a1_6300cc786c6e.slice/crio-4e60fa9c65d0c3ece4c2be0cc26b299ed5c744051ed1ccf0d260f60b8d41d2d1 WatchSource:0}: Error finding container 4e60fa9c65d0c3ece4c2be0cc26b299ed5c744051ed1ccf0d260f60b8d41d2d1: Status 404 returned error can't find the container with id 4e60fa9c65d0c3ece4c2be0cc26b299ed5c744051ed1ccf0d260f60b8d41d2d1
Jul 24 00:14:54.223641 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.223385    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b"]
Jul 24 00:14:54.295101 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.295065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r722z" event={"ID":"1bae97a1-6b3a-4ff1-847b-2df09e067715","Type":"ContainerStarted","Data":"401684c9c097ceea26a28cdaabfad2312199bb4067ead8d43ca812ef3703592c"}
Jul 24 00:14:54.295245 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.295231    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:14:54.296956 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.296922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w86dg" event={"ID":"9da248de-af65-442a-9766-eae338141049","Type":"ContainerStarted","Data":"db2dec35247ce584374602e2f52d42285742160c999184c6446d3466db6b8455"}
Jul 24 00:14:54.298652 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.298569    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kfm7w" event={"ID":"483be78f-e144-4a62-8932-88df4550a1af","Type":"ContainerStarted","Data":"f7bb22bdb0c0cb0ce5cbc0a04c5061dfdda6ca775d3e9e6abeac424da6d4e795"}
Jul 24 00:14:54.299649 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.299602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s" event={"ID":"962628cc-0a5b-43fc-91a1-6300cc786c6e","Type":"ContainerStarted","Data":"4e60fa9c65d0c3ece4c2be0cc26b299ed5c744051ed1ccf0d260f60b8d41d2d1"}
Jul 24 00:14:54.301267 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.301242    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kgsn8" event={"ID":"71f46a43-3c16-45f8-90c4-0f815e6850d0","Type":"ContainerStarted","Data":"9ebcd33ba703e0535be8d6e25fd67cdc69b33c9d24d57642f7feee2c7fa7dc71"}
Jul 24 00:14:54.302588 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.302540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b" event={"ID":"355598f1-a660-4e77-bca5-1b719fd5bc72","Type":"ContainerStarted","Data":"fbe612ca1c66a033e49cbeaf5a998fbb77e2fd5a63279ff89976b173757debf8"}
Jul 24 00:14:54.312923 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.312294    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r722z" podStartSLOduration=67.059316574 podStartE2EDuration="1m14.312279702s" totalImagesPullingTime="7.252963128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:54.311806517 +0000 UTC m=+74.879622669" watchObservedRunningTime="2026-07-24 00:14:54.312279702 +0000 UTC m=+74.880095857"
Jul 24 00:14:54.329958 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.329901    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w86dg" podStartSLOduration=9.252933826 podStartE2EDuration="25.329885361s" totalImagesPullingTime="16.076951535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:14:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:54.327983208 +0000 UTC m=+74.895799386" watchObservedRunningTime="2026-07-24 00:14:54.329885361 +0000 UTC m=+74.897701511"
Jul 24 00:14:54.343202 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:54.343135    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kgsn8" podStartSLOduration=67.299878401 podStartE2EDuration="1m14.343116563s" totalImagesPullingTime="7.043238162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:54.341569203 +0000 UTC m=+74.909385354" watchObservedRunningTime="2026-07-24 00:14:54.343116563 +0000 UTC m=+74.910932712"
Jul 24 00:14:55.312193 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:55.311912    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kfm7w" event={"ID":"483be78f-e144-4a62-8932-88df4550a1af","Type":"ContainerStarted","Data":"69f6a9aefb814b44e4adb53903b88730b417241713a74a116470c2daacfa3634"}
Jul 24 00:14:55.312193 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:55.312138    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:14:55.328845 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:55.328775    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kfm7w" podStartSLOduration=34.940229181 podStartE2EDuration="40.328755178s" totalImagesPullingTime="5.388525997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:55.327577057 +0000 UTC m=+75.895393207" watchObservedRunningTime="2026-07-24 00:14:55.328755178 +0000 UTC m=+75.896571338"
Jul 24 00:14:55.496850 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:55.496800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:55.505271 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:55.505237    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc5d1a31-2a44-4139-b308-bcf393724c12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8x9kb\" (UID: \"cc5d1a31-2a44-4139-b308-bcf393724c12\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:55.706322 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:55.706283    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7hh5x\""
Jul 24 00:14:55.714357 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:55.714319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"
Jul 24 00:14:58.154397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:58.154343    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb"]
Jul 24 00:14:58.157774 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:14:58.157744    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5d1a31_2a44_4139_b308_bcf393724c12.slice/crio-a18f4a9482d15f96c444cd14f8ac96e153c4078a66678b1150625935238ffb43 WatchSource:0}: Error finding container a18f4a9482d15f96c444cd14f8ac96e153c4078a66678b1150625935238ffb43: Status 404 returned error can't find the container with id a18f4a9482d15f96c444cd14f8ac96e153c4078a66678b1150625935238ffb43
Jul 24 00:14:58.325586 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:58.325482    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s" event={"ID":"962628cc-0a5b-43fc-91a1-6300cc786c6e","Type":"ContainerStarted","Data":"ac7338fc2047924352c3e3827a1f250d4c7fbcdb1b90b1b2bc0a990ff21f61bd"}
Jul 24 00:14:58.327685 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:58.327655    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b" event={"ID":"355598f1-a660-4e77-bca5-1b719fd5bc72","Type":"ContainerStarted","Data":"810b1543275d361f3b387d366cb51261d288436392778681f2f8aac6511a8a5c"}
Jul 24 00:14:58.327863 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:58.327692    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b" event={"ID":"355598f1-a660-4e77-bca5-1b719fd5bc72","Type":"ContainerStarted","Data":"a3cd5e11cf89da2149c993101c5280a7692c5020834ddf49333efd0c44d6e34c"}
Jul 24 00:14:58.327863 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:58.327705    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b" event={"ID":"355598f1-a660-4e77-bca5-1b719fd5bc72","Type":"ContainerStarted","Data":"5faec41e8b4b77c527c7a9a72691241473b76a45ddd613a4ab681029bc7d0a38"}
Jul 24 00:14:58.328869 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:58.328849    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb" event={"ID":"cc5d1a31-2a44-4139-b308-bcf393724c12","Type":"ContainerStarted","Data":"a18f4a9482d15f96c444cd14f8ac96e153c4078a66678b1150625935238ffb43"}
Jul 24 00:14:58.340044 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:58.339904    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f5669ffdd-wlm8s" podStartSLOduration=1.533394659 podStartE2EDuration="5.339887374s" totalImagesPullingTime="3.806492715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:58.339713441 +0000 UTC m=+78.907529591" watchObservedRunningTime="2026-07-24 00:14:58.339887374 +0000 UTC m=+78.907703525"
Jul 24 00:14:58.357085 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:14:58.357040    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6695d6c4b5-wrh5b" podStartSLOduration=1.5590903539999998 podStartE2EDuration="5.357024074s" totalImagesPullingTime="3.79793372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:14:58.356176028 +0000 UTC m=+78.923992178" watchObservedRunningTime="2026-07-24 00:14:58.357024074 +0000 UTC m=+78.924840281"
Jul 24 00:15:00.338613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:00.338534    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb" event={"ID":"cc5d1a31-2a44-4139-b308-bcf393724c12","Type":"ContainerStarted","Data":"90ec6edabac791498eb94541eabb96aa2fe2f80a26bba17dbc8761a5b5ecb5d7"}
Jul 24 00:15:00.355194 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:00.355147    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8x9kb" podStartSLOduration=35.559410096 podStartE2EDuration="37.35513212s" totalImagesPullingTime="1.795722024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:15:00.354186688 +0000 UTC m=+80.922002842" watchObservedRunningTime="2026-07-24 00:15:00.35513212 +0000 UTC m=+80.922948301"
Jul 24 00:15:05.319004 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:05.318887    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kfm7w"
Jul 24 00:15:07.903093 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:07.903049    2560 patch_prober.go:28] interesting pod/image-registry-576d988ffd-2kxdz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 24 00:15:07.903499 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:07.903112    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz" podUID="1fa5df10-e395-47e4-a3d8-bea5b049a19f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 00:15:10.885423 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:10.885377    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bvnkw"]
Jul 24 00:15:10.890232 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:10.890206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:10.892562 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:10.892538    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtkhg\""
Jul 24 00:15:10.892690 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:10.892578    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 00:15:10.893271 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:10.893250    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 00:15:10.893271 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:10.893270    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 00:15:10.893413 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:10.893293    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 00:15:11.007961 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.007921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/015bd884-f690-4837-a093-b29203d0fd09-metrics-client-ca\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.008143 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.007971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxf72\" (UniqueName: \"kubernetes.io/projected/015bd884-f690-4837-a093-b29203d0fd09-kube-api-access-xxf72\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.008143 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.008037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-tls\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.008143 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.008098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-textfile\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.008143 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.008140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.008293 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.008170    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/015bd884-f690-4837-a093-b29203d0fd09-sys\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.008293 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.008231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/015bd884-f690-4837-a093-b29203d0fd09-root\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.008293 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.008259    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-accelerators-collector-config\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.008399 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.008294    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-wtmp\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109347 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109534 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109359    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/015bd884-f690-4837-a093-b29203d0fd09-sys\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109534 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/015bd884-f690-4837-a093-b29203d0fd09-root\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109534 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-accelerators-collector-config\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109534 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-wtmp\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109534 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109477    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/015bd884-f690-4837-a093-b29203d0fd09-metrics-client-ca\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109534 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxf72\" (UniqueName: \"kubernetes.io/projected/015bd884-f690-4837-a093-b29203d0fd09-kube-api-access-xxf72\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109534 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/015bd884-f690-4837-a093-b29203d0fd09-sys\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109534 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109526    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/015bd884-f690-4837-a093-b29203d0fd09-root\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109966 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109536    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-tls\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109966 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-textfile\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.109966 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.109668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-wtmp\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.110333 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.110292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/015bd884-f690-4837-a093-b29203d0fd09-metrics-client-ca\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.110467 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.110303    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-accelerators-collector-config\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.110467 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.110405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-textfile\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.111942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.111919    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.112184 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.112167    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/015bd884-f690-4837-a093-b29203d0fd09-node-exporter-tls\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.119256 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.119232    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxf72\" (UniqueName: \"kubernetes.io/projected/015bd884-f690-4837-a093-b29203d0fd09-kube-api-access-xxf72\") pod \"node-exporter-bvnkw\" (UID: \"015bd884-f690-4837-a093-b29203d0fd09\") " pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.201959 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.201918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bvnkw"
Jul 24 00:15:11.212497 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:15:11.212454    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015bd884_f690_4837_a093_b29203d0fd09.slice/crio-c614c71af26bd320a8cff07c691b23500a8b3358fc8351b14d8101e6466ee1a0 WatchSource:0}: Error finding container c614c71af26bd320a8cff07c691b23500a8b3358fc8351b14d8101e6466ee1a0: Status 404 returned error can't find the container with id c614c71af26bd320a8cff07c691b23500a8b3358fc8351b14d8101e6466ee1a0
Jul 24 00:15:11.286968 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.286932    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:15:11.378027 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:11.377992    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bvnkw" event={"ID":"015bd884-f690-4837-a093-b29203d0fd09","Type":"ContainerStarted","Data":"c614c71af26bd320a8cff07c691b23500a8b3358fc8351b14d8101e6466ee1a0"}
Jul 24 00:15:13.392557 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:13.392520    2560 generic.go:356] "Generic (PLEG): container finished" podID="015bd884-f690-4837-a093-b29203d0fd09" containerID="e4b1c8687a9341c6ccf5c51dbc0c835edb88b29e9aa530a106af4175e6be64f9" exitCode=0
Jul 24 00:15:13.393039 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:13.392594    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bvnkw" event={"ID":"015bd884-f690-4837-a093-b29203d0fd09","Type":"ContainerDied","Data":"e4b1c8687a9341c6ccf5c51dbc0c835edb88b29e9aa530a106af4175e6be64f9"}
Jul 24 00:15:14.399163 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:14.399124    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bvnkw" event={"ID":"015bd884-f690-4837-a093-b29203d0fd09","Type":"ContainerStarted","Data":"4caabfbaea6729e62ad05ab916fe896862a3c1028b8ece1f31ab62c25e05c3cc"}
Jul 24 00:15:14.399613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:14.399187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bvnkw" event={"ID":"015bd884-f690-4837-a093-b29203d0fd09","Type":"ContainerStarted","Data":"4bda05f8ee3e0e726e8ccd68c83249039b497c1224d8d11aac506d64a9fa39e8"}
Jul 24 00:15:14.420556 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:14.420500    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bvnkw" podStartSLOduration=2.654179858 podStartE2EDuration="4.42048075s" totalImagesPullingTime="1.686744465s" totalInitContainerRuntime="79.556427ms" isStatefulPod=true podCreationTimestamp="2026-07-24 00:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:15:14.419163241 +0000 UTC m=+94.986979417" watchObservedRunningTime="2026-07-24 00:15:14.42048075 +0000 UTC m=+94.988296901"
Jul 24 00:15:18.395251 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:18.395214    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-576d988ffd-2kxdz"]
Jul 24 00:15:25.315315 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:25.315280    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r722z"
Jul 24 00:15:38.490186 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:38.490152    2560 generic.go:356] "Generic (PLEG): container finished" podID="f7122daf-91a2-4e73-97d5-c1d236cb5a6b" containerID="204551c49bd70e6b533a2ccd35f04b6ce6e3aa577d88a563dda7afa8ffb806de" exitCode=0
Jul 24 00:15:38.490591 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:38.490216    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk" event={"ID":"f7122daf-91a2-4e73-97d5-c1d236cb5a6b","Type":"ContainerDied","Data":"204551c49bd70e6b533a2ccd35f04b6ce6e3aa577d88a563dda7afa8ffb806de"}
Jul 24 00:15:38.490591 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:38.490509    2560 scope.go:117] "RemoveContainer" containerID="204551c49bd70e6b533a2ccd35f04b6ce6e3aa577d88a563dda7afa8ffb806de"
Jul 24 00:15:39.499243 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:39.499207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-4p6jk" event={"ID":"f7122daf-91a2-4e73-97d5-c1d236cb5a6b","Type":"ContainerStarted","Data":"506fc0252bd509935f2968dedbedd362f2082be0779ca6217e7785edf9bf7c30"}
Jul 24 00:15:43.414370 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.414310    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz" podUID="1fa5df10-e395-47e4-a3d8-bea5b049a19f" containerName="registry" containerID="cri-o://7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed" gracePeriod=30
Jul 24 00:15:43.656335 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.656306    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:15:43.737612 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.737571    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-installation-pull-secrets\") pod \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") "
Jul 24 00:15:43.737612 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.737618    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-bound-sa-token\") pod \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") "
Jul 24 00:15:43.737871 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.737645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fa5df10-e395-47e4-a3d8-bea5b049a19f-ca-trust-extracted\") pod \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") "
Jul 24 00:15:43.737871 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.737662    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-certificates\") pod \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") "
Jul 24 00:15:43.737871 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.737684    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls\") pod \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") "
Jul 24 00:15:43.737871 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.737726    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-image-registry-private-configuration\") pod \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") "
Jul 24 00:15:43.737871 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.737753    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgx7h\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-kube-api-access-rgx7h\") pod \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") "
Jul 24 00:15:43.737871 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.737782    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-trusted-ca\") pod \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\" (UID: \"1fa5df10-e395-47e4-a3d8-bea5b049a19f\") "
Jul 24 00:15:43.738374 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.738280    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1fa5df10-e395-47e4-a3d8-bea5b049a19f" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:15:43.738374 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.738324    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1fa5df10-e395-47e4-a3d8-bea5b049a19f" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:15:43.740483 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.740455    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1fa5df10-e395-47e4-a3d8-bea5b049a19f" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:15:43.740596 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.740571    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1fa5df10-e395-47e4-a3d8-bea5b049a19f" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:15:43.740668 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.740634    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1fa5df10-e395-47e4-a3d8-bea5b049a19f" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:15:43.740714 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.740669    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1fa5df10-e395-47e4-a3d8-bea5b049a19f" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:15:43.740764 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.740740    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-kube-api-access-rgx7h" (OuterVolumeSpecName: "kube-api-access-rgx7h") pod "1fa5df10-e395-47e4-a3d8-bea5b049a19f" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f"). InnerVolumeSpecName "kube-api-access-rgx7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:15:43.745600 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.745577    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa5df10-e395-47e4-a3d8-bea5b049a19f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1fa5df10-e395-47e4-a3d8-bea5b049a19f" (UID: "1fa5df10-e395-47e4-a3d8-bea5b049a19f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:15:43.838545 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.838502    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-installation-pull-secrets\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:15:43.838545 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.838536    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-bound-sa-token\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:15:43.838545 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.838547    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fa5df10-e395-47e4-a3d8-bea5b049a19f-ca-trust-extracted\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:15:43.838545 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.838555    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-certificates\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:15:43.838795 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.838566    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-registry-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:15:43.838795 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.838575    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fa5df10-e395-47e4-a3d8-bea5b049a19f-image-registry-private-configuration\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:15:43.838795 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.838583    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgx7h\" (UniqueName: \"kubernetes.io/projected/1fa5df10-e395-47e4-a3d8-bea5b049a19f-kube-api-access-rgx7h\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:15:43.838795 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:43.838591    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fa5df10-e395-47e4-a3d8-bea5b049a19f-trusted-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:15:44.518483 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:44.518443    2560 generic.go:356] "Generic (PLEG): container finished" podID="1fa5df10-e395-47e4-a3d8-bea5b049a19f" containerID="7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed" exitCode=0
Jul 24 00:15:44.518899 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:44.518511    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz"
Jul 24 00:15:44.518899 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:44.518534    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz" event={"ID":"1fa5df10-e395-47e4-a3d8-bea5b049a19f","Type":"ContainerDied","Data":"7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed"}
Jul 24 00:15:44.518899 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:44.518615    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-576d988ffd-2kxdz" event={"ID":"1fa5df10-e395-47e4-a3d8-bea5b049a19f","Type":"ContainerDied","Data":"49aa1fd7985a4d0ce3b20cf6e0c8e2dafcd895775bb32e4974efaf02b22da4c5"}
Jul 24 00:15:44.518899 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:44.518637    2560 scope.go:117] "RemoveContainer" containerID="7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed"
Jul 24 00:15:44.526998 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:44.526980    2560 scope.go:117] "RemoveContainer" containerID="7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed"
Jul 24 00:15:44.527285 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:15:44.527264    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed\": container with ID starting with 7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed not found: ID does not exist" containerID="7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed"
Jul 24 00:15:44.527344 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:44.527294    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed"} err="failed to get container status \"7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed\": rpc error: code = NotFound desc = could not find container \"7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed\": container with ID starting with 7c00b2470bfa44bff0fc3224f36122c7f8e01261d40ae6f6fde45b2396c0d2ed not found: ID does not exist"
Jul 24 00:15:44.535601 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:44.535576    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-576d988ffd-2kxdz"]
Jul 24 00:15:44.540097 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:44.540076    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-576d988ffd-2kxdz"]
Jul 24 00:15:45.966459 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:45.966424    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fa5df10-e395-47e4-a3d8-bea5b049a19f" path="/var/lib/kubelet/pods/1fa5df10-e395-47e4-a3d8-bea5b049a19f/volumes"
Jul 24 00:15:48.533018 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:48.532927    2560 generic.go:356] "Generic (PLEG): container finished" podID="5b5fb683-60bd-483c-a20b-51cd9a0285df" containerID="9cfda213352dee5f86b6483968d88cda4a81a7e07ffb21c0b30e9939d49ab9b9" exitCode=0
Jul 24 00:15:48.533018 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:48.532985    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449" event={"ID":"5b5fb683-60bd-483c-a20b-51cd9a0285df","Type":"ContainerDied","Data":"9cfda213352dee5f86b6483968d88cda4a81a7e07ffb21c0b30e9939d49ab9b9"}
Jul 24 00:15:48.533418 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:48.533333    2560 scope.go:117] "RemoveContainer" containerID="9cfda213352dee5f86b6483968d88cda4a81a7e07ffb21c0b30e9939d49ab9b9"
Jul 24 00:15:49.538370 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:15:49.538336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-jt449" event={"ID":"5b5fb683-60bd-483c-a20b-51cd9a0285df","Type":"ContainerStarted","Data":"f4d6285f6391754390379f547c6d9d1de628f7053b0a2bc4827a0791c42d96db"}
Jul 24 00:18:39.863032 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:39.863002    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 00:18:54.755906 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.755869    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"]
Jul 24 00:18:54.758141 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.756116    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fa5df10-e395-47e4-a3d8-bea5b049a19f" containerName="registry"
Jul 24 00:18:54.758141 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.756126    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fa5df10-e395-47e4-a3d8-bea5b049a19f" containerName="registry"
Jul 24 00:18:54.758141 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.756179    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fa5df10-e395-47e4-a3d8-bea5b049a19f" containerName="registry"
Jul 24 00:18:54.758929 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.758912    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.761516 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.761485    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 00:18:54.761727 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.761708    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b26lx\""
Jul 24 00:18:54.761790 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.761708    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 00:18:54.761920 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.761902    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 00:18:54.762027 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.762010    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 00:18:54.781109 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.781077    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"]
Jul 24 00:18:54.809970 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.809936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km9jr\" (UniqueName: \"kubernetes.io/projected/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-kube-api-access-km9jr\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.810161 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.809977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.810161 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.810010    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-manifests\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.810161 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.810044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.810161 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.810089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-tmp\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.911345 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.911306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-tmp\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.911565 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.911374    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km9jr\" (UniqueName: \"kubernetes.io/projected/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-kube-api-access-km9jr\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.911565 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.911401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.911565 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.911432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-manifests\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.911565 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.911469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.911913 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.911884    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-manifests\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.913899 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.913863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-tmp\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.914008 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.913945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.914049 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.914027    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:54.923275 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:54.923246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km9jr\" (UniqueName: \"kubernetes.io/projected/5a3ded49-8443-40d7-83c2-f14adf4a2cc0-kube-api-access-km9jr\") pod \"opendatahub-operator-controller-manager-6cff767bd6-q7l64\" (UID: \"5a3ded49-8443-40d7-83c2-f14adf4a2cc0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:55.068766 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:55.068661    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:18:55.204259 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:55.204225    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"]
Jul 24 00:18:55.207402 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:18:55.207371    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3ded49_8443_40d7_83c2_f14adf4a2cc0.slice/crio-25bd9e8545c4f506c9c2cb47cd6385dc06af4ddbd0befa2cdad34f9f74f66270 WatchSource:0}: Error finding container 25bd9e8545c4f506c9c2cb47cd6385dc06af4ddbd0befa2cdad34f9f74f66270: Status 404 returned error can't find the container with id 25bd9e8545c4f506c9c2cb47cd6385dc06af4ddbd0befa2cdad34f9f74f66270
Jul 24 00:18:55.209188 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:55.209170    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:18:56.184129 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:56.184087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64" event={"ID":"5a3ded49-8443-40d7-83c2-f14adf4a2cc0","Type":"ContainerStarted","Data":"25bd9e8545c4f506c9c2cb47cd6385dc06af4ddbd0befa2cdad34f9f74f66270"}
Jul 24 00:18:59.196004 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:18:59.195964    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64" event={"ID":"5a3ded49-8443-40d7-83c2-f14adf4a2cc0","Type":"ContainerStarted","Data":"66a273580b56fc9584e8d996b2ec39f3268ff4371ba8b1097df128d073f79829"}
Jul 24 00:19:00.200952 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:00.200916    2560 generic.go:356] "Generic (PLEG): container finished" podID="5a3ded49-8443-40d7-83c2-f14adf4a2cc0" containerID="66a273580b56fc9584e8d996b2ec39f3268ff4371ba8b1097df128d073f79829" exitCode=0
Jul 24 00:19:00.201335 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:00.200995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64" event={"ID":"5a3ded49-8443-40d7-83c2-f14adf4a2cc0","Type":"ContainerDied","Data":"66a273580b56fc9584e8d996b2ec39f3268ff4371ba8b1097df128d073f79829"}
Jul 24 00:19:01.208421 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:01.208372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64" event={"ID":"5a3ded49-8443-40d7-83c2-f14adf4a2cc0","Type":"ContainerStarted","Data":"f4aaff2408b8ece7128211163ca3b66507028640ec239e8282af77664c27f43a"}
Jul 24 00:19:01.209021 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:01.208998    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:19:01.233008 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:01.232954    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64" podStartSLOduration=2.914352406 podStartE2EDuration="7.232937389s" totalImagesPullingTime="3.309350301s" totalInitContainerRuntime="1.009234682s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:01.231774615 +0000 UTC m=+321.799590765" watchObservedRunningTime="2026-07-24 00:19:01.232937389 +0000 UTC m=+321.800753588"
Jul 24 00:19:12.214390 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:12.214357    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cff767bd6-q7l64"
Jul 24 00:19:14.710405 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:14.710369    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wg7tt"]
Jul 24 00:19:14.725130 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:14.725095    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt"
Jul 24 00:19:14.725307 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:14.725236    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wg7tt"]
Jul 24 00:19:14.727472 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:14.727447    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 24 00:19:14.728198 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:14.727572    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9n7p5\""
Jul 24 00:19:14.751625 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:14.751586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mv88\" (UniqueName: \"kubernetes.io/projected/790e6f76-09fa-4d0f-ab8f-6abcd6f748cc-kube-api-access-7mv88\") pod \"odh-model-controller-7998c67dc4-wg7tt\" (UID: \"790e6f76-09fa-4d0f-ab8f-6abcd6f748cc\") " pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt"
Jul 24 00:19:14.751775 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:14.751699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/790e6f76-09fa-4d0f-ab8f-6abcd6f748cc-cert\") pod \"odh-model-controller-7998c67dc4-wg7tt\" (UID: \"790e6f76-09fa-4d0f-ab8f-6abcd6f748cc\") " pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt"
Jul 24 00:19:14.852656 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:14.852620    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mv88\" (UniqueName: \"kubernetes.io/projected/790e6f76-09fa-4d0f-ab8f-6abcd6f748cc-kube-api-access-7mv88\") pod \"odh-model-controller-7998c67dc4-wg7tt\" (UID: \"790e6f76-09fa-4d0f-ab8f-6abcd6f748cc\") " pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt"
Jul 24 00:19:14.852848 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:14.852683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/790e6f76-09fa-4d0f-ab8f-6abcd6f748cc-cert\") pod \"odh-model-controller-7998c67dc4-wg7tt\" (UID: \"790e6f76-09fa-4d0f-ab8f-6abcd6f748cc\") " pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt"
Jul 24 00:19:14.855127 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:14.855100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/790e6f76-09fa-4d0f-ab8f-6abcd6f748cc-cert\") pod \"odh-model-controller-7998c67dc4-wg7tt\" (UID: \"790e6f76-09fa-4d0f-ab8f-6abcd6f748cc\") " pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt"
Jul 24 00:19:14.861211 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:14.861189    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mv88\" (UniqueName: \"kubernetes.io/projected/790e6f76-09fa-4d0f-ab8f-6abcd6f748cc-kube-api-access-7mv88\") pod \"odh-model-controller-7998c67dc4-wg7tt\" (UID: \"790e6f76-09fa-4d0f-ab8f-6abcd6f748cc\") " pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt"
Jul 24 00:19:15.037704 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.037606    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt"
Jul 24 00:19:15.167435 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.167400    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wg7tt"]
Jul 24 00:19:15.170571 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:19:15.170546    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790e6f76_09fa_4d0f_ab8f_6abcd6f748cc.slice/crio-e5de8575b48ebb2d5eb5a40579106d5674ddfecc27eeb1a9e6ca1e861e4d7028 WatchSource:0}: Error finding container e5de8575b48ebb2d5eb5a40579106d5674ddfecc27eeb1a9e6ca1e861e4d7028: Status 404 returned error can't find the container with id e5de8575b48ebb2d5eb5a40579106d5674ddfecc27eeb1a9e6ca1e861e4d7028
Jul 24 00:19:15.263098 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.263060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt" event={"ID":"790e6f76-09fa-4d0f-ab8f-6abcd6f748cc","Type":"ContainerStarted","Data":"e5de8575b48ebb2d5eb5a40579106d5674ddfecc27eeb1a9e6ca1e861e4d7028"}
Jul 24 00:19:15.331481 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.331396    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bhwn4"]
Jul 24 00:19:15.334540 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.334519    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4"
Jul 24 00:19:15.336911 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.336892    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 24 00:19:15.337012 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.336984    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pdtbt\""
Jul 24 00:19:15.345497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.345465    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bhwn4"]
Jul 24 00:19:15.356566 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.356539    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e25170aa-f54f-4cb0-9334-585e5e89d007-tls-certs\") pod \"model-serving-api-54d89bd5fc-bhwn4\" (UID: \"e25170aa-f54f-4cb0-9334-585e5e89d007\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4"
Jul 24 00:19:15.356672 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.356621    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvpr6\" (UniqueName: \"kubernetes.io/projected/e25170aa-f54f-4cb0-9334-585e5e89d007-kube-api-access-dvpr6\") pod \"model-serving-api-54d89bd5fc-bhwn4\" (UID: \"e25170aa-f54f-4cb0-9334-585e5e89d007\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4"
Jul 24 00:19:15.458080 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.458031    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvpr6\" (UniqueName: \"kubernetes.io/projected/e25170aa-f54f-4cb0-9334-585e5e89d007-kube-api-access-dvpr6\") pod \"model-serving-api-54d89bd5fc-bhwn4\" (UID: \"e25170aa-f54f-4cb0-9334-585e5e89d007\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4"
Jul 24 00:19:15.458268 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.458176    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e25170aa-f54f-4cb0-9334-585e5e89d007-tls-certs\") pod \"model-serving-api-54d89bd5fc-bhwn4\" (UID: \"e25170aa-f54f-4cb0-9334-585e5e89d007\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4"
Jul 24 00:19:15.458361 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:19:15.458342    2560 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 24 00:19:15.458436 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:19:15.458425    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e25170aa-f54f-4cb0-9334-585e5e89d007-tls-certs podName:e25170aa-f54f-4cb0-9334-585e5e89d007 nodeName:}" failed. No retries permitted until 2026-07-24 00:19:15.958402454 +0000 UTC m=+336.526218586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e25170aa-f54f-4cb0-9334-585e5e89d007-tls-certs") pod "model-serving-api-54d89bd5fc-bhwn4" (UID: "e25170aa-f54f-4cb0-9334-585e5e89d007") : secret "model-serving-api-tls" not found
Jul 24 00:19:15.467501 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.467469    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvpr6\" (UniqueName: \"kubernetes.io/projected/e25170aa-f54f-4cb0-9334-585e5e89d007-kube-api-access-dvpr6\") pod \"model-serving-api-54d89bd5fc-bhwn4\" (UID: \"e25170aa-f54f-4cb0-9334-585e5e89d007\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4"
Jul 24 00:19:15.962008 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.961939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e25170aa-f54f-4cb0-9334-585e5e89d007-tls-certs\") pod \"model-serving-api-54d89bd5fc-bhwn4\" (UID: \"e25170aa-f54f-4cb0-9334-585e5e89d007\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4"
Jul 24 00:19:15.965034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:15.964979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e25170aa-f54f-4cb0-9334-585e5e89d007-tls-certs\") pod \"model-serving-api-54d89bd5fc-bhwn4\" (UID: \"e25170aa-f54f-4cb0-9334-585e5e89d007\") " pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4"
Jul 24 00:19:16.245362 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:16.245223    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4"
Jul 24 00:19:16.412499 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:16.412288    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bhwn4"]
Jul 24 00:19:16.416055 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:19:16.416016    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25170aa_f54f_4cb0_9334_585e5e89d007.slice/crio-5d2df378a0fb59ead0deda082f03f86c8187808d470d38f8bff6d793e73dc2c8 WatchSource:0}: Error finding container 5d2df378a0fb59ead0deda082f03f86c8187808d470d38f8bff6d793e73dc2c8: Status 404 returned error can't find the container with id 5d2df378a0fb59ead0deda082f03f86c8187808d470d38f8bff6d793e73dc2c8
Jul 24 00:19:17.272753 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:17.272717    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4" event={"ID":"e25170aa-f54f-4cb0-9334-585e5e89d007","Type":"ContainerStarted","Data":"5d2df378a0fb59ead0deda082f03f86c8187808d470d38f8bff6d793e73dc2c8"}
Jul 24 00:19:18.278448 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:18.278410    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt" event={"ID":"790e6f76-09fa-4d0f-ab8f-6abcd6f748cc","Type":"ContainerStarted","Data":"6276bbcb5c0f371a5a6d5568be1e7aa28ded3f43dda4b1b8c3d0ac2c45152f18"}
Jul 24 00:19:18.278919 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:18.278539    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt"
Jul 24 00:19:18.294740 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:18.294685    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt" podStartSLOduration=1.628996339 podStartE2EDuration="4.294667926s" totalImagesPullingTime="2.665671587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:18.292869481 +0000 UTC m=+338.860685634" watchObservedRunningTime="2026-07-24 00:19:18.294667926 +0000 UTC m=+338.862484075"
Jul 24 00:19:19.137261 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.137227    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"]
Jul 24 00:19:19.140492 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.140472    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.143271 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.143240    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 00:19:19.144268 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.144250    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 00:19:19.144367 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.144353    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nln5t\""
Jul 24 00:19:19.144429 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.144377    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 00:19:19.146366 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.146346    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 00:19:19.146854 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.146818    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 00:19:19.161482 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.161452    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"]
Jul 24 00:19:19.188178 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.188116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fb31fa45-e4d2-47a6-b259-9777f096d4fd-manager-config\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.188178 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.188157    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2gzq\" (UniqueName: \"kubernetes.io/projected/fb31fa45-e4d2-47a6-b259-9777f096d4fd-kube-api-access-c2gzq\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.188344 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.188184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb31fa45-e4d2-47a6-b259-9777f096d4fd-metrics-cert\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.188344 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.188263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb31fa45-e4d2-47a6-b259-9777f096d4fd-cert\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.284474 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.284430    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4" event={"ID":"e25170aa-f54f-4cb0-9334-585e5e89d007","Type":"ContainerStarted","Data":"8a378fd73c461abc810b978d62b2afac9fa4eef9bf80d439041be1d7b9119383"}
Jul 24 00:19:19.284899 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.284522    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4"
Jul 24 00:19:19.289108 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.289087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2gzq\" (UniqueName: \"kubernetes.io/projected/fb31fa45-e4d2-47a6-b259-9777f096d4fd-kube-api-access-c2gzq\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.289189 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.289124    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb31fa45-e4d2-47a6-b259-9777f096d4fd-metrics-cert\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.289189 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.289150    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb31fa45-e4d2-47a6-b259-9777f096d4fd-cert\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.289189 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.289181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fb31fa45-e4d2-47a6-b259-9777f096d4fd-manager-config\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.289727 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.289706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fb31fa45-e4d2-47a6-b259-9777f096d4fd-manager-config\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.291807 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.291782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb31fa45-e4d2-47a6-b259-9777f096d4fd-cert\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.291908 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.291782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb31fa45-e4d2-47a6-b259-9777f096d4fd-metrics-cert\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.297405 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.297382    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2gzq\" (UniqueName: \"kubernetes.io/projected/fb31fa45-e4d2-47a6-b259-9777f096d4fd-kube-api-access-c2gzq\") pod \"lws-controller-manager-5b84c6bc85-px7mf\" (UID: \"fb31fa45-e4d2-47a6-b259-9777f096d4fd\") " pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.301706 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.301666    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4" podStartSLOduration=1.801012907 podStartE2EDuration="4.30165188s" totalImagesPullingTime="2.500638973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:19.300857856 +0000 UTC m=+339.868674005" watchObservedRunningTime="2026-07-24 00:19:19.30165188 +0000 UTC m=+339.869468030"
Jul 24 00:19:19.450345 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.450312    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:19.586444 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:19.586332    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"]
Jul 24 00:19:19.589070 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:19:19.589043    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb31fa45_e4d2_47a6_b259_9777f096d4fd.slice/crio-c8e430f25403317c405ed19d18516e1ac56c9698461f1e478162f20c21f3fc86 WatchSource:0}: Error finding container c8e430f25403317c405ed19d18516e1ac56c9698461f1e478162f20c21f3fc86: Status 404 returned error can't find the container with id c8e430f25403317c405ed19d18516e1ac56c9698461f1e478162f20c21f3fc86
Jul 24 00:19:20.289689 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:20.289657    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf" event={"ID":"fb31fa45-e4d2-47a6-b259-9777f096d4fd","Type":"ContainerStarted","Data":"c8e430f25403317c405ed19d18516e1ac56c9698461f1e478162f20c21f3fc86"}
Jul 24 00:19:22.299273 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:22.299181    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf" event={"ID":"fb31fa45-e4d2-47a6-b259-9777f096d4fd","Type":"ContainerStarted","Data":"b800c009e1ac2848fbb331ec70df721062e83968b5f4628af9f7ae753b94d444"}
Jul 24 00:19:22.299273 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:22.299243    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:19:22.315390 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:22.315341    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf" podStartSLOduration=0.908035278 podStartE2EDuration="3.315326535s" totalImagesPullingTime="2.407291257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:22.314172096 +0000 UTC m=+342.881988247" watchObservedRunningTime="2026-07-24 00:19:22.315326535 +0000 UTC m=+342.883142685"
Jul 24 00:19:24.970944 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:24.970904    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"]
Jul 24 00:19:24.974717 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:24.974696    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"
Jul 24 00:19:24.977423 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:24.977400    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9qv2s\""
Jul 24 00:19:24.977742 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:24.977726    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 24 00:19:24.986767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:24.986744    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"]
Jul 24 00:19:24.989963 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:24.989944    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"
Jul 24 00:19:24.992489 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:24.992469    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nr6ll\""
Jul 24 00:19:24.994545 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:24.994525    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 24 00:19:25.004984 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.004961    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"]
Jul 24 00:19:25.031505 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.031472    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"]
Jul 24 00:19:25.042244 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.042207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4995k\" (UniqueName: \"kubernetes.io/projected/9c11c1dc-96a2-4767-ae53-2c7c468bc7d3-kube-api-access-4995k\") pod \"llmisvc-controller-manager-777c6bd7d5-x6c65\" (UID: \"9c11c1dc-96a2-4767-ae53-2c7c468bc7d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"
Jul 24 00:19:25.042391 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.042280    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpr52\" (UniqueName: \"kubernetes.io/projected/e9fdf508-6600-4330-bd5d-4bda72e7d1e5-kube-api-access-vpr52\") pod \"kserve-controller-manager-d6f4b65b8-8pjnq\" (UID: \"e9fdf508-6600-4330-bd5d-4bda72e7d1e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"
Jul 24 00:19:25.042391 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.042325    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9fdf508-6600-4330-bd5d-4bda72e7d1e5-cert\") pod \"kserve-controller-manager-d6f4b65b8-8pjnq\" (UID: \"e9fdf508-6600-4330-bd5d-4bda72e7d1e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"
Jul 24 00:19:25.042391 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.042381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c11c1dc-96a2-4767-ae53-2c7c468bc7d3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-x6c65\" (UID: \"9c11c1dc-96a2-4767-ae53-2c7c468bc7d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"
Jul 24 00:19:25.143426 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.143388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c11c1dc-96a2-4767-ae53-2c7c468bc7d3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-x6c65\" (UID: \"9c11c1dc-96a2-4767-ae53-2c7c468bc7d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"
Jul 24 00:19:25.143632 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.143450    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4995k\" (UniqueName: \"kubernetes.io/projected/9c11c1dc-96a2-4767-ae53-2c7c468bc7d3-kube-api-access-4995k\") pod \"llmisvc-controller-manager-777c6bd7d5-x6c65\" (UID: \"9c11c1dc-96a2-4767-ae53-2c7c468bc7d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"
Jul 24 00:19:25.143632 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.143482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpr52\" (UniqueName: \"kubernetes.io/projected/e9fdf508-6600-4330-bd5d-4bda72e7d1e5-kube-api-access-vpr52\") pod \"kserve-controller-manager-d6f4b65b8-8pjnq\" (UID: \"e9fdf508-6600-4330-bd5d-4bda72e7d1e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"
Jul 24 00:19:25.143632 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.143500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9fdf508-6600-4330-bd5d-4bda72e7d1e5-cert\") pod \"kserve-controller-manager-d6f4b65b8-8pjnq\" (UID: \"e9fdf508-6600-4330-bd5d-4bda72e7d1e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"
Jul 24 00:19:25.146100 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.146064    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c11c1dc-96a2-4767-ae53-2c7c468bc7d3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-x6c65\" (UID: \"9c11c1dc-96a2-4767-ae53-2c7c468bc7d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"
Jul 24 00:19:25.146210 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.146111    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9fdf508-6600-4330-bd5d-4bda72e7d1e5-cert\") pod \"kserve-controller-manager-d6f4b65b8-8pjnq\" (UID: \"e9fdf508-6600-4330-bd5d-4bda72e7d1e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"
Jul 24 00:19:25.151774 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.151724    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4995k\" (UniqueName: \"kubernetes.io/projected/9c11c1dc-96a2-4767-ae53-2c7c468bc7d3-kube-api-access-4995k\") pod \"llmisvc-controller-manager-777c6bd7d5-x6c65\" (UID: \"9c11c1dc-96a2-4767-ae53-2c7c468bc7d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"
Jul 24 00:19:25.151912 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.151890    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpr52\" (UniqueName: \"kubernetes.io/projected/e9fdf508-6600-4330-bd5d-4bda72e7d1e5-kube-api-access-vpr52\") pod \"kserve-controller-manager-d6f4b65b8-8pjnq\" (UID: \"e9fdf508-6600-4330-bd5d-4bda72e7d1e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"
Jul 24 00:19:25.287860 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.287737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"
Jul 24 00:19:25.301053 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.301027    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"
Jul 24 00:19:25.448805 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.448773    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"]
Jul 24 00:19:25.452986 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:19:25.452957    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fdf508_6600_4330_bd5d_4bda72e7d1e5.slice/crio-608ff9cf77bce54836a741573fa58367722b69e2af8d2bcae65c0d41148a20c9 WatchSource:0}: Error finding container 608ff9cf77bce54836a741573fa58367722b69e2af8d2bcae65c0d41148a20c9: Status 404 returned error can't find the container with id 608ff9cf77bce54836a741573fa58367722b69e2af8d2bcae65c0d41148a20c9
Jul 24 00:19:25.463589 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:25.463522    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"]
Jul 24 00:19:25.466260 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:19:25.466238    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9c11c1dc_96a2_4767_ae53_2c7c468bc7d3.slice/crio-c0234c856387f024f1b1d52743c243d8dade963d3310bfc16593fe72d1cfa575 WatchSource:0}: Error finding container c0234c856387f024f1b1d52743c243d8dade963d3310bfc16593fe72d1cfa575: Status 404 returned error can't find the container with id c0234c856387f024f1b1d52743c243d8dade963d3310bfc16593fe72d1cfa575
Jul 24 00:19:26.317708 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:26.317668    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq" event={"ID":"e9fdf508-6600-4330-bd5d-4bda72e7d1e5","Type":"ContainerStarted","Data":"608ff9cf77bce54836a741573fa58367722b69e2af8d2bcae65c0d41148a20c9"}
Jul 24 00:19:26.319287 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:26.319254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65" event={"ID":"9c11c1dc-96a2-4767-ae53-2c7c468bc7d3","Type":"ContainerStarted","Data":"c0234c856387f024f1b1d52743c243d8dade963d3310bfc16593fe72d1cfa575"}
Jul 24 00:19:29.287663 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:29.287579    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wg7tt"
Jul 24 00:19:29.333695 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:29.333658    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65" event={"ID":"9c11c1dc-96a2-4767-ae53-2c7c468bc7d3","Type":"ContainerStarted","Data":"2a4335bcfbafcbb567e1f1bb64cd07d85dc0a6f9c89a0b9218a3ca7a10db9f75"}
Jul 24 00:19:29.333910 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:29.333741    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"
Jul 24 00:19:29.335497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:29.335457    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq" event={"ID":"e9fdf508-6600-4330-bd5d-4bda72e7d1e5","Type":"ContainerStarted","Data":"b97a07bb17222836dc6ac2b430c4ae1bd7c2487af238157cc5bc79e70b76db25"}
Jul 24 00:19:29.335647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:29.335597    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"
Jul 24 00:19:29.351745 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:29.351689    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65" podStartSLOduration=1.833369341 podStartE2EDuration="5.351670604s" totalImagesPullingTime="3.518301263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:29.35050084 +0000 UTC m=+349.918316992" watchObservedRunningTime="2026-07-24 00:19:29.351670604 +0000 UTC m=+349.919486756"
Jul 24 00:19:29.366315 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:29.366254    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq" podStartSLOduration=1.832563778 podStartE2EDuration="5.366235713s" totalImagesPullingTime="3.533671935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:29.365022175 +0000 UTC m=+349.932838327" watchObservedRunningTime="2026-07-24 00:19:29.366235713 +0000 UTC m=+349.934051868"
Jul 24 00:19:30.294485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:30.294455    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-bhwn4"
Jul 24 00:19:33.306059 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:19:33.305962    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b84c6bc85-px7mf"
Jul 24 00:20:00.342959 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:00.342913    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x6c65"
Jul 24 00:20:00.345917 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:00.345898    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pjnq"
Jul 24 00:20:49.075896 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.075857    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf"]
Jul 24 00:20:49.079051 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.079027    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf"
Jul 24 00:20:49.081537 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.081510    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 00:20:49.082313 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.082293    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2f7tz\""
Jul 24 00:20:49.082422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.082331    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 00:20:49.093944 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.093915    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf"]
Jul 24 00:20:49.184531 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.184491    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqnm\" (UniqueName: \"kubernetes.io/projected/ac0ceea5-a2aa-44c4-95d3-bc4862cad4f3-kube-api-access-bkqnm\") pod \"authorino-operator-75cc8d5cb7-gx4jf\" (UID: \"ac0ceea5-a2aa-44c4-95d3-bc4862cad4f3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf"
Jul 24 00:20:49.285153 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.285109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqnm\" (UniqueName: \"kubernetes.io/projected/ac0ceea5-a2aa-44c4-95d3-bc4862cad4f3-kube-api-access-bkqnm\") pod \"authorino-operator-75cc8d5cb7-gx4jf\" (UID: \"ac0ceea5-a2aa-44c4-95d3-bc4862cad4f3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf"
Jul 24 00:20:49.294510 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.294478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqnm\" (UniqueName: \"kubernetes.io/projected/ac0ceea5-a2aa-44c4-95d3-bc4862cad4f3-kube-api-access-bkqnm\") pod \"authorino-operator-75cc8d5cb7-gx4jf\" (UID: \"ac0ceea5-a2aa-44c4-95d3-bc4862cad4f3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf"
Jul 24 00:20:49.389508 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.389409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf"
Jul 24 00:20:49.526324 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.526290    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf"]
Jul 24 00:20:49.529969 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:20:49.529943    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0ceea5_a2aa_44c4_95d3_bc4862cad4f3.slice/crio-08f2ed13136aea7ff22c29f77676e50b686bebbf9cda7235ee6a2ad610b82627 WatchSource:0}: Error finding container 08f2ed13136aea7ff22c29f77676e50b686bebbf9cda7235ee6a2ad610b82627: Status 404 returned error can't find the container with id 08f2ed13136aea7ff22c29f77676e50b686bebbf9cda7235ee6a2ad610b82627
Jul 24 00:20:49.663064 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:49.662970    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf" event={"ID":"ac0ceea5-a2aa-44c4-95d3-bc4862cad4f3","Type":"ContainerStarted","Data":"08f2ed13136aea7ff22c29f77676e50b686bebbf9cda7235ee6a2ad610b82627"}
Jul 24 00:20:52.681369 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:52.681318    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf" event={"ID":"ac0ceea5-a2aa-44c4-95d3-bc4862cad4f3","Type":"ContainerStarted","Data":"bc285dfbbcdb15b258af71ff8f582bfe8fc653616c57ccd2c5886e08eba24e02"}
Jul 24 00:20:52.681747 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:52.681412    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf"
Jul 24 00:20:52.701540 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:20:52.701486    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf" podStartSLOduration=1.176797643 podStartE2EDuration="3.701470305s" totalImagesPullingTime="2.524672662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:20:52.701102768 +0000 UTC m=+433.268918919" watchObservedRunningTime="2026-07-24 00:20:52.701470305 +0000 UTC m=+433.269286454"
Jul 24 00:21:03.687968 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:03.687882    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gx4jf"
Jul 24 00:21:56.387926 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.387885    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gql5w"]
Jul 24 00:21:56.391280 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.391258    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:21:56.393486 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.393462    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 00:21:56.393586 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.393465    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sbjbq\""
Jul 24 00:21:56.398932 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.398906    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gql5w"]
Jul 24 00:21:56.418154 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.418126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7dbce1a-5bc1-425e-b827-0a4b45415c71-config-file\") pod \"limitador-limitador-64c94fd64f-gql5w\" (UID: \"b7dbce1a-5bc1-425e-b827-0a4b45415c71\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:21:56.418259 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.418162    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j542m\" (UniqueName: \"kubernetes.io/projected/b7dbce1a-5bc1-425e-b827-0a4b45415c71-kube-api-access-j542m\") pod \"limitador-limitador-64c94fd64f-gql5w\" (UID: \"b7dbce1a-5bc1-425e-b827-0a4b45415c71\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:21:56.482974 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.482941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gql5w"]
Jul 24 00:21:56.519139 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.519109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7dbce1a-5bc1-425e-b827-0a4b45415c71-config-file\") pod \"limitador-limitador-64c94fd64f-gql5w\" (UID: \"b7dbce1a-5bc1-425e-b827-0a4b45415c71\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:21:56.519139 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.519141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j542m\" (UniqueName: \"kubernetes.io/projected/b7dbce1a-5bc1-425e-b827-0a4b45415c71-kube-api-access-j542m\") pod \"limitador-limitador-64c94fd64f-gql5w\" (UID: \"b7dbce1a-5bc1-425e-b827-0a4b45415c71\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:21:56.519723 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.519703    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7dbce1a-5bc1-425e-b827-0a4b45415c71-config-file\") pod \"limitador-limitador-64c94fd64f-gql5w\" (UID: \"b7dbce1a-5bc1-425e-b827-0a4b45415c71\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:21:56.526671 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.526637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j542m\" (UniqueName: \"kubernetes.io/projected/b7dbce1a-5bc1-425e-b827-0a4b45415c71-kube-api-access-j542m\") pod \"limitador-limitador-64c94fd64f-gql5w\" (UID: \"b7dbce1a-5bc1-425e-b827-0a4b45415c71\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:21:56.701621 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.701583    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:21:56.833535 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.833500    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gql5w"]
Jul 24 00:21:56.836701 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:21:56.836675    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7dbce1a_5bc1_425e_b827_0a4b45415c71.slice/crio-b814c6610196703ea8216a77f25de7689586f1f56aec09750430a9cfd4f1acf1 WatchSource:0}: Error finding container b814c6610196703ea8216a77f25de7689586f1f56aec09750430a9cfd4f1acf1: Status 404 returned error can't find the container with id b814c6610196703ea8216a77f25de7689586f1f56aec09750430a9cfd4f1acf1
Jul 24 00:21:56.944253 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:56.944219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w" event={"ID":"b7dbce1a-5bc1-425e-b827-0a4b45415c71","Type":"ContainerStarted","Data":"b814c6610196703ea8216a77f25de7689586f1f56aec09750430a9cfd4f1acf1"}
Jul 24 00:21:57.416858 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:57.416801    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-d7tzv"]
Jul 24 00:21:57.421095 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:57.421073    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d7tzv"
Jul 24 00:21:57.423226 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:57.423205    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8vf4k\""
Jul 24 00:21:57.426312 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:57.426291    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d7tzv"]
Jul 24 00:21:57.527854 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:57.527793    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdkvs\" (UniqueName: \"kubernetes.io/projected/d425cd9e-5f01-47f7-a9eb-5afb36fc65f4-kube-api-access-bdkvs\") pod \"authorino-75d556bd77-d7tzv\" (UID: \"d425cd9e-5f01-47f7-a9eb-5afb36fc65f4\") " pod="kuadrant-system/authorino-75d556bd77-d7tzv"
Jul 24 00:21:57.628925 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:57.628885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdkvs\" (UniqueName: \"kubernetes.io/projected/d425cd9e-5f01-47f7-a9eb-5afb36fc65f4-kube-api-access-bdkvs\") pod \"authorino-75d556bd77-d7tzv\" (UID: \"d425cd9e-5f01-47f7-a9eb-5afb36fc65f4\") " pod="kuadrant-system/authorino-75d556bd77-d7tzv"
Jul 24 00:21:57.640457 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:57.640421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdkvs\" (UniqueName: \"kubernetes.io/projected/d425cd9e-5f01-47f7-a9eb-5afb36fc65f4-kube-api-access-bdkvs\") pod \"authorino-75d556bd77-d7tzv\" (UID: \"d425cd9e-5f01-47f7-a9eb-5afb36fc65f4\") " pod="kuadrant-system/authorino-75d556bd77-d7tzv"
Jul 24 00:21:57.733762 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:57.733728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d7tzv"
Jul 24 00:21:57.923615 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:57.923575    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d7tzv"]
Jul 24 00:21:57.927234 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:21:57.927207    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd425cd9e_5f01_47f7_a9eb_5afb36fc65f4.slice/crio-cfcb1e651c3645db08f7c7a574bfa7a881bc211f382b3b05eb108361a42bc542 WatchSource:0}: Error finding container cfcb1e651c3645db08f7c7a574bfa7a881bc211f382b3b05eb108361a42bc542: Status 404 returned error can't find the container with id cfcb1e651c3645db08f7c7a574bfa7a881bc211f382b3b05eb108361a42bc542
Jul 24 00:21:57.952081 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:21:57.952017    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d7tzv" event={"ID":"d425cd9e-5f01-47f7-a9eb-5afb36fc65f4","Type":"ContainerStarted","Data":"cfcb1e651c3645db08f7c7a574bfa7a881bc211f382b3b05eb108361a42bc542"}
Jul 24 00:22:01.977573 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:01.977536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d7tzv" event={"ID":"d425cd9e-5f01-47f7-a9eb-5afb36fc65f4","Type":"ContainerStarted","Data":"c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7"}
Jul 24 00:22:01.979078 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:01.979046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w" event={"ID":"b7dbce1a-5bc1-425e-b827-0a4b45415c71","Type":"ContainerStarted","Data":"aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb"}
Jul 24 00:22:01.979183 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:01.979110    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:22:01.990569 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:01.990496    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-d7tzv" podStartSLOduration=1.43236502 podStartE2EDuration="4.990480154s" totalImagesPullingTime="3.558115134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:22:01.989955738 +0000 UTC m=+502.557771887" watchObservedRunningTime="2026-07-24 00:22:01.990480154 +0000 UTC m=+502.558296305"
Jul 24 00:22:02.007560 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:02.007512    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w" podStartSLOduration=1.366376351 podStartE2EDuration="6.007495569s" totalImagesPullingTime="4.641119218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:22:02.005897162 +0000 UTC m=+502.573713314" watchObservedRunningTime="2026-07-24 00:22:02.007495569 +0000 UTC m=+502.575311719"
Jul 24 00:22:11.373629 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:11.373586    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gql5w"]
Jul 24 00:22:11.374100 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:11.373807    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w" podUID="b7dbce1a-5bc1-425e-b827-0a4b45415c71" containerName="limitador" containerID="cri-o://aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb" gracePeriod=30
Jul 24 00:22:11.374758 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:11.374729    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:22:11.953723 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:11.953696    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:22:12.022150 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.022115    2560 generic.go:356] "Generic (PLEG): container finished" podID="b7dbce1a-5bc1-425e-b827-0a4b45415c71" containerID="aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb" exitCode=0
Jul 24 00:22:12.022314 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.022171    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w"
Jul 24 00:22:12.022314 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.022189    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w" event={"ID":"b7dbce1a-5bc1-425e-b827-0a4b45415c71","Type":"ContainerDied","Data":"aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb"}
Jul 24 00:22:12.022314 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.022231    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gql5w" event={"ID":"b7dbce1a-5bc1-425e-b827-0a4b45415c71","Type":"ContainerDied","Data":"b814c6610196703ea8216a77f25de7689586f1f56aec09750430a9cfd4f1acf1"}
Jul 24 00:22:12.022314 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.022252    2560 scope.go:117] "RemoveContainer" containerID="aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb"
Jul 24 00:22:12.030417 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.030399    2560 scope.go:117] "RemoveContainer" containerID="aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb"
Jul 24 00:22:12.030690 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:22:12.030672    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb\": container with ID starting with aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb not found: ID does not exist" containerID="aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb"
Jul 24 00:22:12.030740 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.030700    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb"} err="failed to get container status \"aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb\": rpc error: code = NotFound desc = could not find container \"aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb\": container with ID starting with aa093be8f21d133e326d99057d18c1703e2317ddbad911e516cc1dc651b154bb not found: ID does not exist"
Jul 24 00:22:12.050975 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.050949    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7dbce1a-5bc1-425e-b827-0a4b45415c71-config-file\") pod \"b7dbce1a-5bc1-425e-b827-0a4b45415c71\" (UID: \"b7dbce1a-5bc1-425e-b827-0a4b45415c71\") "
Jul 24 00:22:12.051100 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.051053    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j542m\" (UniqueName: \"kubernetes.io/projected/b7dbce1a-5bc1-425e-b827-0a4b45415c71-kube-api-access-j542m\") pod \"b7dbce1a-5bc1-425e-b827-0a4b45415c71\" (UID: \"b7dbce1a-5bc1-425e-b827-0a4b45415c71\") "
Jul 24 00:22:12.051336 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.051311    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7dbce1a-5bc1-425e-b827-0a4b45415c71-config-file" (OuterVolumeSpecName: "config-file") pod "b7dbce1a-5bc1-425e-b827-0a4b45415c71" (UID: "b7dbce1a-5bc1-425e-b827-0a4b45415c71"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:22:12.053186 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.053162    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7dbce1a-5bc1-425e-b827-0a4b45415c71-kube-api-access-j542m" (OuterVolumeSpecName: "kube-api-access-j542m") pod "b7dbce1a-5bc1-425e-b827-0a4b45415c71" (UID: "b7dbce1a-5bc1-425e-b827-0a4b45415c71"). InnerVolumeSpecName "kube-api-access-j542m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:22:12.152254 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.152219    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j542m\" (UniqueName: \"kubernetes.io/projected/b7dbce1a-5bc1-425e-b827-0a4b45415c71-kube-api-access-j542m\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:22:12.152254 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.152252    2560 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7dbce1a-5bc1-425e-b827-0a4b45415c71-config-file\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:22:12.344444 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.344412    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gql5w"]
Jul 24 00:22:12.350186 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:12.350159    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gql5w"]
Jul 24 00:22:13.008656 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.008623    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"]
Jul 24 00:22:13.009046 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.008962    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7dbce1a-5bc1-425e-b827-0a4b45415c71" containerName="limitador"
Jul 24 00:22:13.009046 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.008973    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7dbce1a-5bc1-425e-b827-0a4b45415c71" containerName="limitador"
Jul 24 00:22:13.009046 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.009025    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7dbce1a-5bc1-425e-b827-0a4b45415c71" containerName="limitador"
Jul 24 00:22:13.013518 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.013494    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"
Jul 24 00:22:13.016517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.016495    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 24 00:22:13.016896 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.016878    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-7n899\""
Jul 24 00:22:13.017453 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.017436    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 00:22:13.017511 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.017480    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 00:22:13.021689 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.021668    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"]
Jul 24 00:22:13.160076 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.160035    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ef89d7fd-4e6b-4660-b06d-edbd560b2ed4-data\") pod \"postgres-69fb4594cf-km5mh\" (UID: \"ef89d7fd-4e6b-4660-b06d-edbd560b2ed4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"
Jul 24 00:22:13.160076 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.160075    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kz9p\" (UniqueName: \"kubernetes.io/projected/ef89d7fd-4e6b-4660-b06d-edbd560b2ed4-kube-api-access-8kz9p\") pod \"postgres-69fb4594cf-km5mh\" (UID: \"ef89d7fd-4e6b-4660-b06d-edbd560b2ed4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"
Jul 24 00:22:13.260806 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.260717    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ef89d7fd-4e6b-4660-b06d-edbd560b2ed4-data\") pod \"postgres-69fb4594cf-km5mh\" (UID: \"ef89d7fd-4e6b-4660-b06d-edbd560b2ed4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"
Jul 24 00:22:13.260806 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.260758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kz9p\" (UniqueName: \"kubernetes.io/projected/ef89d7fd-4e6b-4660-b06d-edbd560b2ed4-kube-api-access-8kz9p\") pod \"postgres-69fb4594cf-km5mh\" (UID: \"ef89d7fd-4e6b-4660-b06d-edbd560b2ed4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"
Jul 24 00:22:13.261086 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.261064    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ef89d7fd-4e6b-4660-b06d-edbd560b2ed4-data\") pod \"postgres-69fb4594cf-km5mh\" (UID: \"ef89d7fd-4e6b-4660-b06d-edbd560b2ed4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"
Jul 24 00:22:13.269318 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.269292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kz9p\" (UniqueName: \"kubernetes.io/projected/ef89d7fd-4e6b-4660-b06d-edbd560b2ed4-kube-api-access-8kz9p\") pod \"postgres-69fb4594cf-km5mh\" (UID: \"ef89d7fd-4e6b-4660-b06d-edbd560b2ed4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"
Jul 24 00:22:13.325288 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.325258    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"
Jul 24 00:22:13.453542 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.453510    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"]
Jul 24 00:22:13.457734 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:22:13.457709    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef89d7fd_4e6b_4660_b06d_edbd560b2ed4.slice/crio-79df41076b41160688ad0a7ef43813382b99ee158139e34f8fdd1e4fcd5211ae WatchSource:0}: Error finding container 79df41076b41160688ad0a7ef43813382b99ee158139e34f8fdd1e4fcd5211ae: Status 404 returned error can't find the container with id 79df41076b41160688ad0a7ef43813382b99ee158139e34f8fdd1e4fcd5211ae
Jul 24 00:22:13.966667 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:13.966630    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7dbce1a-5bc1-425e-b827-0a4b45415c71" path="/var/lib/kubelet/pods/b7dbce1a-5bc1-425e-b827-0a4b45415c71/volumes"
Jul 24 00:22:14.032678 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:14.032642    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh" event={"ID":"ef89d7fd-4e6b-4660-b06d-edbd560b2ed4","Type":"ContainerStarted","Data":"79df41076b41160688ad0a7ef43813382b99ee158139e34f8fdd1e4fcd5211ae"}
Jul 24 00:22:19.057952 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:19.057911    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh" event={"ID":"ef89d7fd-4e6b-4660-b06d-edbd560b2ed4","Type":"ContainerStarted","Data":"292db7dd328b932737c16f467284b4db8a3b71334ed67249ec1f55dc082efd8e"}
Jul 24 00:22:19.058331 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:19.058041    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"
Jul 24 00:22:19.072851 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:19.072785    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh" podStartSLOduration=2.202484423 podStartE2EDuration="7.072770657s" totalImagesPullingTime="4.870286234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:22:19.071856359 +0000 UTC m=+519.639672513" watchObservedRunningTime="2026-07-24 00:22:19.072770657 +0000 UTC m=+519.640586806"
Jul 24 00:22:25.090087 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:25.090052    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-km5mh"
Jul 24 00:22:25.691668 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:25.691623    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-76rxl"]
Jul 24 00:22:25.697766 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:25.697735    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-76rxl"
Jul 24 00:22:25.702644 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:25.702616    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-76rxl"]
Jul 24 00:22:25.875767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:25.875726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sjcd\" (UniqueName: \"kubernetes.io/projected/cec36fd1-a4d7-4495-bda3-29363ed4b0f6-kube-api-access-9sjcd\") pod \"authorino-5d4b67c446-76rxl\" (UID: \"cec36fd1-a4d7-4495-bda3-29363ed4b0f6\") " pod="kuadrant-system/authorino-5d4b67c446-76rxl"
Jul 24 00:22:25.941734 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:25.941639    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-76rxl"]
Jul 24 00:22:25.941972 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:22:25.941947    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9sjcd], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-76rxl" podUID="cec36fd1-a4d7-4495-bda3-29363ed4b0f6"
Jul 24 00:22:25.967283 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:25.967251    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f5596495-2l6fj"]
Jul 24 00:22:25.971167 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:25.971144    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f5596495-2l6fj"
Jul 24 00:22:25.975585 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:25.975556    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f5596495-2l6fj"]
Jul 24 00:22:25.976196 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:25.976174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sjcd\" (UniqueName: \"kubernetes.io/projected/cec36fd1-a4d7-4495-bda3-29363ed4b0f6-kube-api-access-9sjcd\") pod \"authorino-5d4b67c446-76rxl\" (UID: \"cec36fd1-a4d7-4495-bda3-29363ed4b0f6\") " pod="kuadrant-system/authorino-5d4b67c446-76rxl"
Jul 24 00:22:25.983592 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:25.983568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sjcd\" (UniqueName: \"kubernetes.io/projected/cec36fd1-a4d7-4495-bda3-29363ed4b0f6-kube-api-access-9sjcd\") pod \"authorino-5d4b67c446-76rxl\" (UID: \"cec36fd1-a4d7-4495-bda3-29363ed4b0f6\") " pod="kuadrant-system/authorino-5d4b67c446-76rxl"
Jul 24 00:22:26.076684 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.076640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vndvd\" (UniqueName: \"kubernetes.io/projected/8e9f9cc5-288c-49bb-8ac7-838972a745f0-kube-api-access-vndvd\") pod \"authorino-f5596495-2l6fj\" (UID: \"8e9f9cc5-288c-49bb-8ac7-838972a745f0\") " pod="kuadrant-system/authorino-f5596495-2l6fj"
Jul 24 00:22:26.086092 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.086061    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-76rxl"
Jul 24 00:22:26.093739 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.093711    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-76rxl"
Jul 24 00:22:26.146219 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.146180    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f5596495-2l6fj"]
Jul 24 00:22:26.146480 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:22:26.146458    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vndvd], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-f5596495-2l6fj" podUID="8e9f9cc5-288c-49bb-8ac7-838972a745f0"
Jul 24 00:22:26.174858 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.174805    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-577769c4d7-hz7rf"]
Jul 24 00:22:26.177219 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.177171    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sjcd\" (UniqueName: \"kubernetes.io/projected/cec36fd1-a4d7-4495-bda3-29363ed4b0f6-kube-api-access-9sjcd\") pod \"cec36fd1-a4d7-4495-bda3-29363ed4b0f6\" (UID: \"cec36fd1-a4d7-4495-bda3-29363ed4b0f6\") "
Jul 24 00:22:26.177330 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.177314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndvd\" (UniqueName: \"kubernetes.io/projected/8e9f9cc5-288c-49bb-8ac7-838972a745f0-kube-api-access-vndvd\") pod \"authorino-f5596495-2l6fj\" (UID: \"8e9f9cc5-288c-49bb-8ac7-838972a745f0\") " pod="kuadrant-system/authorino-f5596495-2l6fj"
Jul 24 00:22:26.178598 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.178580    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-577769c4d7-hz7rf"
Jul 24 00:22:26.179295 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.179273    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec36fd1-a4d7-4495-bda3-29363ed4b0f6-kube-api-access-9sjcd" (OuterVolumeSpecName: "kube-api-access-9sjcd") pod "cec36fd1-a4d7-4495-bda3-29363ed4b0f6" (UID: "cec36fd1-a4d7-4495-bda3-29363ed4b0f6"). InnerVolumeSpecName "kube-api-access-9sjcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:22:26.180811 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.180795    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 24 00:22:26.185590 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.185568    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-577769c4d7-hz7rf"]
Jul 24 00:22:26.188292 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.188270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vndvd\" (UniqueName: \"kubernetes.io/projected/8e9f9cc5-288c-49bb-8ac7-838972a745f0-kube-api-access-vndvd\") pod \"authorino-f5596495-2l6fj\" (UID: \"8e9f9cc5-288c-49bb-8ac7-838972a745f0\") " pod="kuadrant-system/authorino-f5596495-2l6fj"
Jul 24 00:22:26.277928 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.277894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq7lr\" (UniqueName: \"kubernetes.io/projected/9f975053-1a46-48d5-b394-c6cad6d20afc-kube-api-access-wq7lr\") pod \"authorino-577769c4d7-hz7rf\" (UID: \"9f975053-1a46-48d5-b394-c6cad6d20afc\") " pod="kuadrant-system/authorino-577769c4d7-hz7rf"
Jul 24 00:22:26.277928 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.277928    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f975053-1a46-48d5-b394-c6cad6d20afc-tls-cert\") pod \"authorino-577769c4d7-hz7rf\" (UID: \"9f975053-1a46-48d5-b394-c6cad6d20afc\") " pod="kuadrant-system/authorino-577769c4d7-hz7rf"
Jul 24 00:22:26.278111 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.277997    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sjcd\" (UniqueName: \"kubernetes.io/projected/cec36fd1-a4d7-4495-bda3-29363ed4b0f6-kube-api-access-9sjcd\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:22:26.379069 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.379029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq7lr\" (UniqueName: \"kubernetes.io/projected/9f975053-1a46-48d5-b394-c6cad6d20afc-kube-api-access-wq7lr\") pod \"authorino-577769c4d7-hz7rf\" (UID: \"9f975053-1a46-48d5-b394-c6cad6d20afc\") " pod="kuadrant-system/authorino-577769c4d7-hz7rf"
Jul 24 00:22:26.379069 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.379068    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f975053-1a46-48d5-b394-c6cad6d20afc-tls-cert\") pod \"authorino-577769c4d7-hz7rf\" (UID: \"9f975053-1a46-48d5-b394-c6cad6d20afc\") " pod="kuadrant-system/authorino-577769c4d7-hz7rf"
Jul 24 00:22:26.381750 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.381717    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f975053-1a46-48d5-b394-c6cad6d20afc-tls-cert\") pod \"authorino-577769c4d7-hz7rf\" (UID: \"9f975053-1a46-48d5-b394-c6cad6d20afc\") " pod="kuadrant-system/authorino-577769c4d7-hz7rf"
Jul 24 00:22:26.387557 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.387534    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq7lr\" (UniqueName: \"kubernetes.io/projected/9f975053-1a46-48d5-b394-c6cad6d20afc-kube-api-access-wq7lr\") pod \"authorino-577769c4d7-hz7rf\" (UID: \"9f975053-1a46-48d5-b394-c6cad6d20afc\") " pod="kuadrant-system/authorino-577769c4d7-hz7rf"
Jul 24 00:22:26.501475 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.501392    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-577769c4d7-hz7rf"
Jul 24 00:22:26.632700 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:26.632673    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-577769c4d7-hz7rf"]
Jul 24 00:22:26.635357 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:22:26.635338    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f975053_1a46_48d5_b394_c6cad6d20afc.slice/crio-e6222932dfc1776760440e0f366b3615f8e9a36c1fdd717b92cff4dd0c076ab3 WatchSource:0}: Error finding container e6222932dfc1776760440e0f366b3615f8e9a36c1fdd717b92cff4dd0c076ab3: Status 404 returned error can't find the container with id e6222932dfc1776760440e0f366b3615f8e9a36c1fdd717b92cff4dd0c076ab3
Jul 24 00:22:27.090906 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:27.090868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-577769c4d7-hz7rf" event={"ID":"9f975053-1a46-48d5-b394-c6cad6d20afc","Type":"ContainerStarted","Data":"e6222932dfc1776760440e0f366b3615f8e9a36c1fdd717b92cff4dd0c076ab3"}
Jul 24 00:22:27.090906 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:27.090892    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-76rxl"
Jul 24 00:22:27.091151 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:27.090901    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f5596495-2l6fj"
Jul 24 00:22:27.099920 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:27.099900    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f5596495-2l6fj"
Jul 24 00:22:27.127923 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:27.127888    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-76rxl"]
Jul 24 00:22:27.131327 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:27.131297    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-76rxl"]
Jul 24 00:22:27.186714 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:27.186681    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vndvd\" (UniqueName: \"kubernetes.io/projected/8e9f9cc5-288c-49bb-8ac7-838972a745f0-kube-api-access-vndvd\") pod \"8e9f9cc5-288c-49bb-8ac7-838972a745f0\" (UID: \"8e9f9cc5-288c-49bb-8ac7-838972a745f0\") "
Jul 24 00:22:27.188792 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:27.188763    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9f9cc5-288c-49bb-8ac7-838972a745f0-kube-api-access-vndvd" (OuterVolumeSpecName: "kube-api-access-vndvd") pod "8e9f9cc5-288c-49bb-8ac7-838972a745f0" (UID: "8e9f9cc5-288c-49bb-8ac7-838972a745f0"). InnerVolumeSpecName "kube-api-access-vndvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:22:27.287686 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:27.287653    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vndvd\" (UniqueName: \"kubernetes.io/projected/8e9f9cc5-288c-49bb-8ac7-838972a745f0-kube-api-access-vndvd\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:22:27.967543 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:27.967510    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cec36fd1-a4d7-4495-bda3-29363ed4b0f6" path="/var/lib/kubelet/pods/cec36fd1-a4d7-4495-bda3-29363ed4b0f6/volumes"
Jul 24 00:22:28.097175 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:28.097133    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-577769c4d7-hz7rf" event={"ID":"9f975053-1a46-48d5-b394-c6cad6d20afc","Type":"ContainerStarted","Data":"8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f"}
Jul 24 00:22:28.097350 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:28.097225    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f5596495-2l6fj"
Jul 24 00:22:28.114354 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:28.114303    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-577769c4d7-hz7rf" podStartSLOduration=1.536961552 podStartE2EDuration="2.114287981s" totalImagesPullingTime="577.326429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:22:28.113108756 +0000 UTC m=+528.680924907" watchObservedRunningTime="2026-07-24 00:22:28.114287981 +0000 UTC m=+528.682104127"
Jul 24 00:22:28.136051 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:28.136017    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f5596495-2l6fj"]
Jul 24 00:22:28.142982 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:28.142950    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f5596495-2l6fj"]
Jul 24 00:22:28.144349 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:28.144324    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d7tzv"]
Jul 24 00:22:28.144533 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:28.144511    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-d7tzv" podUID="d425cd9e-5f01-47f7-a9eb-5afb36fc65f4" containerName="authorino" containerID="cri-o://c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7" gracePeriod=30
Jul 24 00:22:28.421352 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:28.421323    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d7tzv"
Jul 24 00:22:28.597065 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:28.596964    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdkvs\" (UniqueName: \"kubernetes.io/projected/d425cd9e-5f01-47f7-a9eb-5afb36fc65f4-kube-api-access-bdkvs\") pod \"d425cd9e-5f01-47f7-a9eb-5afb36fc65f4\" (UID: \"d425cd9e-5f01-47f7-a9eb-5afb36fc65f4\") "
Jul 24 00:22:28.599089 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:28.599055    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d425cd9e-5f01-47f7-a9eb-5afb36fc65f4-kube-api-access-bdkvs" (OuterVolumeSpecName: "kube-api-access-bdkvs") pod "d425cd9e-5f01-47f7-a9eb-5afb36fc65f4" (UID: "d425cd9e-5f01-47f7-a9eb-5afb36fc65f4"). InnerVolumeSpecName "kube-api-access-bdkvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:22:28.697705 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:28.697657    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdkvs\" (UniqueName: \"kubernetes.io/projected/d425cd9e-5f01-47f7-a9eb-5afb36fc65f4-kube-api-access-bdkvs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:22:29.103020 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:29.102983    2560 generic.go:356] "Generic (PLEG): container finished" podID="d425cd9e-5f01-47f7-a9eb-5afb36fc65f4" containerID="c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7" exitCode=0
Jul 24 00:22:29.103194 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:29.103030    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d7tzv"
Jul 24 00:22:29.103194 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:29.103061    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d7tzv" event={"ID":"d425cd9e-5f01-47f7-a9eb-5afb36fc65f4","Type":"ContainerDied","Data":"c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7"}
Jul 24 00:22:29.103194 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:29.103103    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d7tzv" event={"ID":"d425cd9e-5f01-47f7-a9eb-5afb36fc65f4","Type":"ContainerDied","Data":"cfcb1e651c3645db08f7c7a574bfa7a881bc211f382b3b05eb108361a42bc542"}
Jul 24 00:22:29.103194 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:29.103120    2560 scope.go:117] "RemoveContainer" containerID="c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7"
Jul 24 00:22:29.111609 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:29.111585    2560 scope.go:117] "RemoveContainer" containerID="c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7"
Jul 24 00:22:29.111892 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:22:29.111872    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7\": container with ID starting with c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7 not found: ID does not exist" containerID="c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7"
Jul 24 00:22:29.111960 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:29.111901    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7"} err="failed to get container status \"c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7\": rpc error: code = NotFound desc = could not find container \"c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7\": container with ID starting with c91cd904d5dc0c6153cc82ed56353fd0bcc9db626242a9eb9d268d6c2fb00bc7 not found: ID does not exist"
Jul 24 00:22:29.123140 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:29.123116    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d7tzv"]
Jul 24 00:22:29.125164 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:29.125143    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d7tzv"]
Jul 24 00:22:29.967364 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:29.967329    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e9f9cc5-288c-49bb-8ac7-838972a745f0" path="/var/lib/kubelet/pods/8e9f9cc5-288c-49bb-8ac7-838972a745f0/volumes"
Jul 24 00:22:29.967577 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:29.967561    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d425cd9e-5f01-47f7-a9eb-5afb36fc65f4" path="/var/lib/kubelet/pods/d425cd9e-5f01-47f7-a9eb-5afb36fc65f4/volumes"
Jul 24 00:22:33.258072 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.257993    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d76449b94-dxzmp"]
Jul 24 00:22:33.258425 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.258304    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d425cd9e-5f01-47f7-a9eb-5afb36fc65f4" containerName="authorino"
Jul 24 00:22:33.258425 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.258315    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d425cd9e-5f01-47f7-a9eb-5afb36fc65f4" containerName="authorino"
Jul 24 00:22:33.258425 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.258376    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d425cd9e-5f01-47f7-a9eb-5afb36fc65f4" containerName="authorino"
Jul 24 00:22:33.273941 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.273906    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d76449b94-dxzmp"]
Jul 24 00:22:33.274106 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.274025    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:22:33.277414 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.277389    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-n6cg2\""
Jul 24 00:22:33.277555 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.277418    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 00:22:33.277555 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.277392    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 00:22:33.437146 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.437109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdvh\" (UniqueName: \"kubernetes.io/projected/41297a22-32d0-4876-b364-45f7d44b1209-kube-api-access-dxdvh\") pod \"maas-controller-d76449b94-dxzmp\" (UID: \"41297a22-32d0-4876-b364-45f7d44b1209\") " pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:22:33.437146 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.437147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41297a22-32d0-4876-b364-45f7d44b1209-webhook-cert\") pod \"maas-controller-d76449b94-dxzmp\" (UID: \"41297a22-32d0-4876-b364-45f7d44b1209\") " pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:22:33.537967 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.537876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdvh\" (UniqueName: \"kubernetes.io/projected/41297a22-32d0-4876-b364-45f7d44b1209-kube-api-access-dxdvh\") pod \"maas-controller-d76449b94-dxzmp\" (UID: \"41297a22-32d0-4876-b364-45f7d44b1209\") " pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:22:33.537967 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.537917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41297a22-32d0-4876-b364-45f7d44b1209-webhook-cert\") pod \"maas-controller-d76449b94-dxzmp\" (UID: \"41297a22-32d0-4876-b364-45f7d44b1209\") " pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:22:33.540297 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.540268    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41297a22-32d0-4876-b364-45f7d44b1209-webhook-cert\") pod \"maas-controller-d76449b94-dxzmp\" (UID: \"41297a22-32d0-4876-b364-45f7d44b1209\") " pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:22:33.544882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.544855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdvh\" (UniqueName: \"kubernetes.io/projected/41297a22-32d0-4876-b364-45f7d44b1209-kube-api-access-dxdvh\") pod \"maas-controller-d76449b94-dxzmp\" (UID: \"41297a22-32d0-4876-b364-45f7d44b1209\") " pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:22:33.584694 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.584656    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:22:33.711161 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:33.711119    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d76449b94-dxzmp"]
Jul 24 00:22:33.714090 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:22:33.714058    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41297a22_32d0_4876_b364_45f7d44b1209.slice/crio-783c583232db5e7cef179fd0842dd7486fd1e8346768dfbf1124cf12cda2b800 WatchSource:0}: Error finding container 783c583232db5e7cef179fd0842dd7486fd1e8346768dfbf1124cf12cda2b800: Status 404 returned error can't find the container with id 783c583232db5e7cef179fd0842dd7486fd1e8346768dfbf1124cf12cda2b800
Jul 24 00:22:34.132961 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:34.132920    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d76449b94-dxzmp" event={"ID":"41297a22-32d0-4876-b364-45f7d44b1209","Type":"ContainerStarted","Data":"783c583232db5e7cef179fd0842dd7486fd1e8346768dfbf1124cf12cda2b800"}
Jul 24 00:22:36.144631 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:36.144589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d76449b94-dxzmp" event={"ID":"41297a22-32d0-4876-b364-45f7d44b1209","Type":"ContainerStarted","Data":"d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6"}
Jul 24 00:22:36.145112 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:36.144725    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:22:36.171625 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:36.171566    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d76449b94-dxzmp" podStartSLOduration=0.865267217 podStartE2EDuration="3.171546643s" totalImagesPullingTime="2.306279426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:22:36.169760508 +0000 UTC m=+536.737576659" watchObservedRunningTime="2026-07-24 00:22:36.171546643 +0000 UTC m=+536.739362794"
Jul 24 00:22:47.153390 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:22:47.153356    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:23:07.629622 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.629582    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"]
Jul 24 00:23:07.632890 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.632871    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:07.635742 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.635720    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 24 00:23:07.635886 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.635770    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7thnf\""
Jul 24 00:23:07.643367 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.643334    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"]
Jul 24 00:23:07.709343 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.709306    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce3e0420-3fc5-4faa-be8a-5a59be301692-maas-api-tls\") pod \"maas-api-5d6c55ddd9-vwtrh\" (UID: \"ce3e0420-3fc5-4faa-be8a-5a59be301692\") " pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:07.709527 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.709369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdn2\" (UniqueName: \"kubernetes.io/projected/ce3e0420-3fc5-4faa-be8a-5a59be301692-kube-api-access-cgdn2\") pod \"maas-api-5d6c55ddd9-vwtrh\" (UID: \"ce3e0420-3fc5-4faa-be8a-5a59be301692\") " pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:07.709527 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.709413    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce3e0420-3fc5-4faa-be8a-5a59be301692-openshift-service-ca\") pod \"maas-api-5d6c55ddd9-vwtrh\" (UID: \"ce3e0420-3fc5-4faa-be8a-5a59be301692\") " pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:07.810117 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.810078    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce3e0420-3fc5-4faa-be8a-5a59be301692-maas-api-tls\") pod \"maas-api-5d6c55ddd9-vwtrh\" (UID: \"ce3e0420-3fc5-4faa-be8a-5a59be301692\") " pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:07.810294 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.810131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdn2\" (UniqueName: \"kubernetes.io/projected/ce3e0420-3fc5-4faa-be8a-5a59be301692-kube-api-access-cgdn2\") pod \"maas-api-5d6c55ddd9-vwtrh\" (UID: \"ce3e0420-3fc5-4faa-be8a-5a59be301692\") " pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:07.810294 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.810169    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce3e0420-3fc5-4faa-be8a-5a59be301692-openshift-service-ca\") pod \"maas-api-5d6c55ddd9-vwtrh\" (UID: \"ce3e0420-3fc5-4faa-be8a-5a59be301692\") " pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:07.810294 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:23:07.810243    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 24 00:23:07.810429 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:23:07.810334    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce3e0420-3fc5-4faa-be8a-5a59be301692-maas-api-tls podName:ce3e0420-3fc5-4faa-be8a-5a59be301692 nodeName:}" failed. No retries permitted until 2026-07-24 00:23:08.310311038 +0000 UTC m=+568.878127169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce3e0420-3fc5-4faa-be8a-5a59be301692-maas-api-tls") pod "maas-api-5d6c55ddd9-vwtrh" (UID: "ce3e0420-3fc5-4faa-be8a-5a59be301692") : secret "maas-api-serving-cert" not found
Jul 24 00:23:07.810898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.810878    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce3e0420-3fc5-4faa-be8a-5a59be301692-openshift-service-ca\") pod \"maas-api-5d6c55ddd9-vwtrh\" (UID: \"ce3e0420-3fc5-4faa-be8a-5a59be301692\") " pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:07.818592 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:07.818561    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdn2\" (UniqueName: \"kubernetes.io/projected/ce3e0420-3fc5-4faa-be8a-5a59be301692-kube-api-access-cgdn2\") pod \"maas-api-5d6c55ddd9-vwtrh\" (UID: \"ce3e0420-3fc5-4faa-be8a-5a59be301692\") " pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:08.184505 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.184469    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-nw6g8"]
Jul 24 00:23:08.187929 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.187911    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8"
Jul 24 00:23:08.190337 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.190305    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-llzxl\""
Jul 24 00:23:08.190467 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.190373    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 00:23:08.190467 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.190417    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 24 00:23:08.190713 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.190696    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 00:23:08.202217 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.202193    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-nw6g8"]
Jul 24 00:23:08.314578 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.314542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce3e0420-3fc5-4faa-be8a-5a59be301692-maas-api-tls\") pod \"maas-api-5d6c55ddd9-vwtrh\" (UID: \"ce3e0420-3fc5-4faa-be8a-5a59be301692\") " pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:08.314740 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.314600    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695ae2f8-1dcd-4555-b5b1-f683cacb89e4-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-nw6g8\" (UID: \"695ae2f8-1dcd-4555-b5b1-f683cacb89e4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8"
Jul 24 00:23:08.314740 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.314636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hxl\" (UniqueName: \"kubernetes.io/projected/695ae2f8-1dcd-4555-b5b1-f683cacb89e4-kube-api-access-t5hxl\") pod \"payload-processing-5cc6cd444b-nw6g8\" (UID: \"695ae2f8-1dcd-4555-b5b1-f683cacb89e4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8"
Jul 24 00:23:08.316959 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.316932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce3e0420-3fc5-4faa-be8a-5a59be301692-maas-api-tls\") pod \"maas-api-5d6c55ddd9-vwtrh\" (UID: \"ce3e0420-3fc5-4faa-be8a-5a59be301692\") " pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:08.415877 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.415840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695ae2f8-1dcd-4555-b5b1-f683cacb89e4-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-nw6g8\" (UID: \"695ae2f8-1dcd-4555-b5b1-f683cacb89e4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8"
Jul 24 00:23:08.416079 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.415892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hxl\" (UniqueName: \"kubernetes.io/projected/695ae2f8-1dcd-4555-b5b1-f683cacb89e4-kube-api-access-t5hxl\") pod \"payload-processing-5cc6cd444b-nw6g8\" (UID: \"695ae2f8-1dcd-4555-b5b1-f683cacb89e4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8"
Jul 24 00:23:08.416634 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.416612    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695ae2f8-1dcd-4555-b5b1-f683cacb89e4-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-nw6g8\" (UID: \"695ae2f8-1dcd-4555-b5b1-f683cacb89e4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8"
Jul 24 00:23:08.423735 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.423704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hxl\" (UniqueName: \"kubernetes.io/projected/695ae2f8-1dcd-4555-b5b1-f683cacb89e4-kube-api-access-t5hxl\") pod \"payload-processing-5cc6cd444b-nw6g8\" (UID: \"695ae2f8-1dcd-4555-b5b1-f683cacb89e4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8"
Jul 24 00:23:08.499590 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.499553    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8"
Jul 24 00:23:08.544838 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.544793    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:08.640995 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.640907    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-nw6g8"]
Jul 24 00:23:08.644112 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:23:08.644084    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695ae2f8_1dcd_4555_b5b1_f683cacb89e4.slice/crio-0b6c55382f36721c61660e82fed7bf46eed619be9ed94f3db5b71f5fd9418bd3 WatchSource:0}: Error finding container 0b6c55382f36721c61660e82fed7bf46eed619be9ed94f3db5b71f5fd9418bd3: Status 404 returned error can't find the container with id 0b6c55382f36721c61660e82fed7bf46eed619be9ed94f3db5b71f5fd9418bd3
Jul 24 00:23:08.692885 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:08.692854    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"]
Jul 24 00:23:08.695362 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:23:08.695332    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3e0420_3fc5_4faa_be8a_5a59be301692.slice/crio-16ac9b5cc24c50e5cdfe2e078d2eb3ec26d712b3c66d75d880ea90caea22e86a WatchSource:0}: Error finding container 16ac9b5cc24c50e5cdfe2e078d2eb3ec26d712b3c66d75d880ea90caea22e86a: Status 404 returned error can't find the container with id 16ac9b5cc24c50e5cdfe2e078d2eb3ec26d712b3c66d75d880ea90caea22e86a
Jul 24 00:23:09.290184 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:09.290147    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh" event={"ID":"ce3e0420-3fc5-4faa-be8a-5a59be301692","Type":"ContainerStarted","Data":"16ac9b5cc24c50e5cdfe2e078d2eb3ec26d712b3c66d75d880ea90caea22e86a"}
Jul 24 00:23:09.291795 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:09.291684    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8" event={"ID":"695ae2f8-1dcd-4555-b5b1-f683cacb89e4","Type":"ContainerStarted","Data":"0b6c55382f36721c61660e82fed7bf46eed619be9ed94f3db5b71f5fd9418bd3"}
Jul 24 00:23:12.307178 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:12.307138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh" event={"ID":"ce3e0420-3fc5-4faa-be8a-5a59be301692","Type":"ContainerStarted","Data":"9ace7c0f17b8a28cd48b9afb00bbdefe85fb0d417bf9324c9d3996134fc5afbf"}
Jul 24 00:23:12.307644 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:12.307226    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:12.308815 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:12.308790    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8" event={"ID":"695ae2f8-1dcd-4555-b5b1-f683cacb89e4","Type":"ContainerStarted","Data":"d32f559298ac1783ef16772a1fb7948c252fd4b170cae1e1f4dcb1749db3c094"}
Jul 24 00:23:12.308927 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:12.308865    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8"
Jul 24 00:23:12.325843 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:12.325788    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh" podStartSLOduration=2.651605912 podStartE2EDuration="5.325776717s" totalImagesPullingTime="2.674170805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:23:12.323470143 +0000 UTC m=+572.891286292" watchObservedRunningTime="2026-07-24 00:23:12.325776717 +0000 UTC m=+572.893592867"
Jul 24 00:23:12.341567 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:12.341520    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8" podStartSLOduration=1.616804405 podStartE2EDuration="4.341506108s" totalImagesPullingTime="2.724701703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:23:12.339652077 +0000 UTC m=+572.907468239" watchObservedRunningTime="2026-07-24 00:23:12.341506108 +0000 UTC m=+572.909322258"
Jul 24 00:23:18.319090 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:18.319060    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d6c55ddd9-vwtrh"
Jul 24 00:23:23.316009 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:23.315972    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-nw6g8"
Jul 24 00:23:38.492220 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.492183    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"]
Jul 24 00:23:38.495533 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.495510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.499103 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.499073    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bk7h5\""
Jul 24 00:23:38.499240 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.499128    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 00:23:38.499240 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.499074    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 00:23:38.499240 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.499078    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 24 00:23:38.505274 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.505252    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"]
Jul 24 00:23:38.671850 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.671797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.672010 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.671864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxktc\" (UniqueName: \"kubernetes.io/projected/fdae42e0-3b15-4eeb-a909-7ae17865ee24-kube-api-access-kxktc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.672010 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.671903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdae42e0-3b15-4eeb-a909-7ae17865ee24-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.672010 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.671998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.672110 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.672060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.672110 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.672082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.772929 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.772816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.772929 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.772897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.772929 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.772925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.773191 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.772966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.773191 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.773069    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxktc\" (UniqueName: \"kubernetes.io/projected/fdae42e0-3b15-4eeb-a909-7ae17865ee24-kube-api-access-kxktc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.773191 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.773126    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdae42e0-3b15-4eeb-a909-7ae17865ee24-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.773336 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.773294    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.773336 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.773311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.773444 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.773402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.775277 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.775244    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdae42e0-3b15-4eeb-a909-7ae17865ee24-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.775438 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.775422    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdae42e0-3b15-4eeb-a909-7ae17865ee24-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.781100 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.781077    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxktc\" (UniqueName: \"kubernetes.io/projected/fdae42e0-3b15-4eeb-a909-7ae17865ee24-kube-api-access-kxktc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f47jj\" (UID: \"fdae42e0-3b15-4eeb-a909-7ae17865ee24\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.809069 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.809046    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:38.942374 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:38.942347    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"]
Jul 24 00:23:38.944953 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:23:38.944928    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdae42e0_3b15_4eeb_a909_7ae17865ee24.slice/crio-317c47f6fb81cd10440c20962da778f391a68433fd57e18e964ae2164994d7ae WatchSource:0}: Error finding container 317c47f6fb81cd10440c20962da778f391a68433fd57e18e964ae2164994d7ae: Status 404 returned error can't find the container with id 317c47f6fb81cd10440c20962da778f391a68433fd57e18e964ae2164994d7ae
Jul 24 00:23:39.432357 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:39.432313    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj" event={"ID":"fdae42e0-3b15-4eeb-a909-7ae17865ee24","Type":"ContainerStarted","Data":"317c47f6fb81cd10440c20962da778f391a68433fd57e18e964ae2164994d7ae"}
Jul 24 00:23:40.439266 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:40.439181    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj" event={"ID":"fdae42e0-3b15-4eeb-a909-7ae17865ee24","Type":"ContainerStarted","Data":"7ac7527273238c3ee68a301264180713615442c90501621e7fd3ea0891aad124"}
Jul 24 00:23:40.457928 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:40.457873    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj" podStartSLOduration=1.318250116 podStartE2EDuration="2.457856114s" totalImagesPullingTime="1.139605998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:23:40.45663527 +0000 UTC m=+601.024451431" watchObservedRunningTime="2026-07-24 00:23:40.457856114 +0000 UTC m=+601.025672264"
Jul 24 00:23:48.809772 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:48.809713    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:48.810189 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:48.809805    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:48.823283 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:48.823251    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:23:49.489901 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:23:49.489875    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f47jj"
Jul 24 00:25:19.855114 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:19.855078    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7df697bf54-5rxg7"]
Jul 24 00:25:19.858413 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:19.858393    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7df697bf54-5rxg7"
Jul 24 00:25:19.865553 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:19.865524    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7df697bf54-5rxg7"]
Jul 24 00:25:19.950367 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:19.950331    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bb06662a-e767-4fc3-a09b-b4eb104b2fbb-tls-cert\") pod \"authorino-7df697bf54-5rxg7\" (UID: \"bb06662a-e767-4fc3-a09b-b4eb104b2fbb\") " pod="kuadrant-system/authorino-7df697bf54-5rxg7"
Jul 24 00:25:19.950536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:19.950396    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx5hh\" (UniqueName: \"kubernetes.io/projected/bb06662a-e767-4fc3-a09b-b4eb104b2fbb-kube-api-access-sx5hh\") pod \"authorino-7df697bf54-5rxg7\" (UID: \"bb06662a-e767-4fc3-a09b-b4eb104b2fbb\") " pod="kuadrant-system/authorino-7df697bf54-5rxg7"
Jul 24 00:25:20.051856 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.051802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx5hh\" (UniqueName: \"kubernetes.io/projected/bb06662a-e767-4fc3-a09b-b4eb104b2fbb-kube-api-access-sx5hh\") pod \"authorino-7df697bf54-5rxg7\" (UID: \"bb06662a-e767-4fc3-a09b-b4eb104b2fbb\") " pod="kuadrant-system/authorino-7df697bf54-5rxg7"
Jul 24 00:25:20.052023 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.051954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bb06662a-e767-4fc3-a09b-b4eb104b2fbb-tls-cert\") pod \"authorino-7df697bf54-5rxg7\" (UID: \"bb06662a-e767-4fc3-a09b-b4eb104b2fbb\") " pod="kuadrant-system/authorino-7df697bf54-5rxg7"
Jul 24 00:25:20.054351 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.054327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bb06662a-e767-4fc3-a09b-b4eb104b2fbb-tls-cert\") pod \"authorino-7df697bf54-5rxg7\" (UID: \"bb06662a-e767-4fc3-a09b-b4eb104b2fbb\") " pod="kuadrant-system/authorino-7df697bf54-5rxg7"
Jul 24 00:25:20.059584 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.059560    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx5hh\" (UniqueName: \"kubernetes.io/projected/bb06662a-e767-4fc3-a09b-b4eb104b2fbb-kube-api-access-sx5hh\") pod \"authorino-7df697bf54-5rxg7\" (UID: \"bb06662a-e767-4fc3-a09b-b4eb104b2fbb\") " pod="kuadrant-system/authorino-7df697bf54-5rxg7"
Jul 24 00:25:20.168640 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.168551    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7df697bf54-5rxg7"
Jul 24 00:25:20.302611 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.302574    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7df697bf54-5rxg7"]
Jul 24 00:25:20.305823 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:25:20.305797    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb06662a_e767_4fc3_a09b_b4eb104b2fbb.slice/crio-585b3fbe599278204b6bb740d3860a485268153a0c65bcde3262766c24578583 WatchSource:0}: Error finding container 585b3fbe599278204b6bb740d3860a485268153a0c65bcde3262766c24578583: Status 404 returned error can't find the container with id 585b3fbe599278204b6bb740d3860a485268153a0c65bcde3262766c24578583
Jul 24 00:25:20.307147 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.307130    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:25:20.891748 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.891700    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7df697bf54-5rxg7" event={"ID":"bb06662a-e767-4fc3-a09b-b4eb104b2fbb","Type":"ContainerStarted","Data":"ac7814f099c724a8a12d40c206d6df13a417e4aba811b78e7aaa17000a4ef99e"}
Jul 24 00:25:20.891748 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.891748    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7df697bf54-5rxg7" event={"ID":"bb06662a-e767-4fc3-a09b-b4eb104b2fbb","Type":"ContainerStarted","Data":"585b3fbe599278204b6bb740d3860a485268153a0c65bcde3262766c24578583"}
Jul 24 00:25:20.910949 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.910849    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7df697bf54-5rxg7" podStartSLOduration=1.55903703 podStartE2EDuration="1.910808652s" totalImagesPullingTime="351.771622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:25:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:25:20.9101145 +0000 UTC m=+701.477930689" watchObservedRunningTime="2026-07-24 00:25:20.910808652 +0000 UTC m=+701.478624801"
Jul 24 00:25:20.940292 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.940252    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-577769c4d7-hz7rf"]
Jul 24 00:25:20.940518 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:20.940489    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-577769c4d7-hz7rf" podUID="9f975053-1a46-48d5-b394-c6cad6d20afc" containerName="authorino" containerID="cri-o://8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f" gracePeriod=30
Jul 24 00:25:21.205286 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.205258    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-577769c4d7-hz7rf"
Jul 24 00:25:21.262273 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.262238    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq7lr\" (UniqueName: \"kubernetes.io/projected/9f975053-1a46-48d5-b394-c6cad6d20afc-kube-api-access-wq7lr\") pod \"9f975053-1a46-48d5-b394-c6cad6d20afc\" (UID: \"9f975053-1a46-48d5-b394-c6cad6d20afc\") "
Jul 24 00:25:21.262466 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.262310    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f975053-1a46-48d5-b394-c6cad6d20afc-tls-cert\") pod \"9f975053-1a46-48d5-b394-c6cad6d20afc\" (UID: \"9f975053-1a46-48d5-b394-c6cad6d20afc\") "
Jul 24 00:25:21.264644 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.264607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f975053-1a46-48d5-b394-c6cad6d20afc-kube-api-access-wq7lr" (OuterVolumeSpecName: "kube-api-access-wq7lr") pod "9f975053-1a46-48d5-b394-c6cad6d20afc" (UID: "9f975053-1a46-48d5-b394-c6cad6d20afc"). InnerVolumeSpecName "kube-api-access-wq7lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:25:21.273269 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.273230    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f975053-1a46-48d5-b394-c6cad6d20afc-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9f975053-1a46-48d5-b394-c6cad6d20afc" (UID: "9f975053-1a46-48d5-b394-c6cad6d20afc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:25:21.363405 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.363365    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq7lr\" (UniqueName: \"kubernetes.io/projected/9f975053-1a46-48d5-b394-c6cad6d20afc-kube-api-access-wq7lr\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:25:21.363405 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.363396    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f975053-1a46-48d5-b394-c6cad6d20afc-tls-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:25:21.898002 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.897961    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f975053-1a46-48d5-b394-c6cad6d20afc" containerID="8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f" exitCode=0
Jul 24 00:25:21.898404 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.898009    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-577769c4d7-hz7rf"
Jul 24 00:25:21.898404 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.898048    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-577769c4d7-hz7rf" event={"ID":"9f975053-1a46-48d5-b394-c6cad6d20afc","Type":"ContainerDied","Data":"8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f"}
Jul 24 00:25:21.898404 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.898093    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-577769c4d7-hz7rf" event={"ID":"9f975053-1a46-48d5-b394-c6cad6d20afc","Type":"ContainerDied","Data":"e6222932dfc1776760440e0f366b3615f8e9a36c1fdd717b92cff4dd0c076ab3"}
Jul 24 00:25:21.898404 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.898117    2560 scope.go:117] "RemoveContainer" containerID="8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f"
Jul 24 00:25:21.907584 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.907564    2560 scope.go:117] "RemoveContainer" containerID="8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f"
Jul 24 00:25:21.907908 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:25:21.907889    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f\": container with ID starting with 8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f not found: ID does not exist" containerID="8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f"
Jul 24 00:25:21.907986 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.907917    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f"} err="failed to get container status \"8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f\": rpc error: code = NotFound desc = could not find container \"8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f\": container with ID starting with 8ab5399668f9c52b00a13489d3b3a7e4729a8c3fde30bd83e4c015cecb6dc47f not found: ID does not exist"
Jul 24 00:25:21.934689 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.934659    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-577769c4d7-hz7rf"]
Jul 24 00:25:21.947099 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.947070    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-577769c4d7-hz7rf"]
Jul 24 00:25:21.966682 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:25:21.966656    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f975053-1a46-48d5-b394-c6cad6d20afc" path="/var/lib/kubelet/pods/9f975053-1a46-48d5-b394-c6cad6d20afc/volumes"
Jul 24 00:26:09.484451 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:09.484406    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d76449b94-dxzmp"]
Jul 24 00:26:09.485078 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:09.484715    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d76449b94-dxzmp" podUID="41297a22-32d0-4876-b364-45f7d44b1209" containerName="manager" containerID="cri-o://d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6" gracePeriod=30
Jul 24 00:26:09.733196 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:09.733172    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:26:09.786639 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:09.786543    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxdvh\" (UniqueName: \"kubernetes.io/projected/41297a22-32d0-4876-b364-45f7d44b1209-kube-api-access-dxdvh\") pod \"41297a22-32d0-4876-b364-45f7d44b1209\" (UID: \"41297a22-32d0-4876-b364-45f7d44b1209\") "
Jul 24 00:26:09.786639 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:09.786605    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41297a22-32d0-4876-b364-45f7d44b1209-webhook-cert\") pod \"41297a22-32d0-4876-b364-45f7d44b1209\" (UID: \"41297a22-32d0-4876-b364-45f7d44b1209\") "
Jul 24 00:26:09.788798 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:09.788762    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41297a22-32d0-4876-b364-45f7d44b1209-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "41297a22-32d0-4876-b364-45f7d44b1209" (UID: "41297a22-32d0-4876-b364-45f7d44b1209"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:26:09.788937 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:09.788875    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41297a22-32d0-4876-b364-45f7d44b1209-kube-api-access-dxdvh" (OuterVolumeSpecName: "kube-api-access-dxdvh") pod "41297a22-32d0-4876-b364-45f7d44b1209" (UID: "41297a22-32d0-4876-b364-45f7d44b1209"). InnerVolumeSpecName "kube-api-access-dxdvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:26:09.887708 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:09.887667    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41297a22-32d0-4876-b364-45f7d44b1209-webhook-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:26:09.887708 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:09.887702    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxdvh\" (UniqueName: \"kubernetes.io/projected/41297a22-32d0-4876-b364-45f7d44b1209-kube-api-access-dxdvh\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:26:10.121451 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:10.121349    2560 generic.go:356] "Generic (PLEG): container finished" podID="41297a22-32d0-4876-b364-45f7d44b1209" containerID="d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6" exitCode=0
Jul 24 00:26:10.121451 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:10.121415    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d76449b94-dxzmp"
Jul 24 00:26:10.121737 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:10.121434    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d76449b94-dxzmp" event={"ID":"41297a22-32d0-4876-b364-45f7d44b1209","Type":"ContainerDied","Data":"d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6"}
Jul 24 00:26:10.121737 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:10.121506    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d76449b94-dxzmp" event={"ID":"41297a22-32d0-4876-b364-45f7d44b1209","Type":"ContainerDied","Data":"783c583232db5e7cef179fd0842dd7486fd1e8346768dfbf1124cf12cda2b800"}
Jul 24 00:26:10.121737 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:10.121535    2560 scope.go:117] "RemoveContainer" containerID="d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6"
Jul 24 00:26:10.130078 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:10.130060    2560 scope.go:117] "RemoveContainer" containerID="d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6"
Jul 24 00:26:10.130343 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:26:10.130322    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6\": container with ID starting with d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6 not found: ID does not exist" containerID="d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6"
Jul 24 00:26:10.130401 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:10.130351    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6"} err="failed to get container status \"d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6\": rpc error: code = NotFound desc = could not find container \"d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6\": container with ID starting with d273ba4907dbbd8b62601cb850ca85c0aebd8811d4a702a667971587f40c3ce6 not found: ID does not exist"
Jul 24 00:26:10.137425 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:10.137401    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d76449b94-dxzmp"]
Jul 24 00:26:10.142493 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:10.142473    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d76449b94-dxzmp"]
Jul 24 00:26:11.967386 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:26:11.967351    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41297a22-32d0-4876-b364-45f7d44b1209" path="/var/lib/kubelet/pods/41297a22-32d0-4876-b364-45f7d44b1209/volumes"
Jul 24 00:27:10.966875 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.966773    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d76449b94-4tmcr"]
Jul 24 00:27:10.967315 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.967112    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f975053-1a46-48d5-b394-c6cad6d20afc" containerName="authorino"
Jul 24 00:27:10.967315 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.967124    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f975053-1a46-48d5-b394-c6cad6d20afc" containerName="authorino"
Jul 24 00:27:10.967315 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.967147    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41297a22-32d0-4876-b364-45f7d44b1209" containerName="manager"
Jul 24 00:27:10.967315 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.967153    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="41297a22-32d0-4876-b364-45f7d44b1209" containerName="manager"
Jul 24 00:27:10.967315 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.967208    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f975053-1a46-48d5-b394-c6cad6d20afc" containerName="authorino"
Jul 24 00:27:10.967315 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.967215    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="41297a22-32d0-4876-b364-45f7d44b1209" containerName="manager"
Jul 24 00:27:10.970148 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.970130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:10.972768 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.972748    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-n6cg2\""
Jul 24 00:27:10.973003 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.972989    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 00:27:10.973746 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.973731    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 00:27:10.980254 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:10.980223    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d76449b94-4tmcr"]
Jul 24 00:27:11.008318 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:11.008274    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/717409be-067d-4092-a201-0a1fe27a20ed-webhook-cert\") pod \"maas-controller-d76449b94-4tmcr\" (UID: \"717409be-067d-4092-a201-0a1fe27a20ed\") " pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:11.008499 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:11.008334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qmh2\" (UniqueName: \"kubernetes.io/projected/717409be-067d-4092-a201-0a1fe27a20ed-kube-api-access-6qmh2\") pod \"maas-controller-d76449b94-4tmcr\" (UID: \"717409be-067d-4092-a201-0a1fe27a20ed\") " pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:11.109566 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:11.109526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/717409be-067d-4092-a201-0a1fe27a20ed-webhook-cert\") pod \"maas-controller-d76449b94-4tmcr\" (UID: \"717409be-067d-4092-a201-0a1fe27a20ed\") " pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:11.109734 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:11.109585    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qmh2\" (UniqueName: \"kubernetes.io/projected/717409be-067d-4092-a201-0a1fe27a20ed-kube-api-access-6qmh2\") pod \"maas-controller-d76449b94-4tmcr\" (UID: \"717409be-067d-4092-a201-0a1fe27a20ed\") " pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:11.112077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:11.112047    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/717409be-067d-4092-a201-0a1fe27a20ed-webhook-cert\") pod \"maas-controller-d76449b94-4tmcr\" (UID: \"717409be-067d-4092-a201-0a1fe27a20ed\") " pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:11.118283 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:11.118259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qmh2\" (UniqueName: \"kubernetes.io/projected/717409be-067d-4092-a201-0a1fe27a20ed-kube-api-access-6qmh2\") pod \"maas-controller-d76449b94-4tmcr\" (UID: \"717409be-067d-4092-a201-0a1fe27a20ed\") " pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:11.280753 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:11.280661    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:11.415255 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:11.415230    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d76449b94-4tmcr"]
Jul 24 00:27:11.417905 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:27:11.417873    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717409be_067d_4092_a201_0a1fe27a20ed.slice/crio-48ce4d6b882cf95d2420d08766f08b997429314e89d5d73c969002fd3a871f3a WatchSource:0}: Error finding container 48ce4d6b882cf95d2420d08766f08b997429314e89d5d73c969002fd3a871f3a: Status 404 returned error can't find the container with id 48ce4d6b882cf95d2420d08766f08b997429314e89d5d73c969002fd3a871f3a
Jul 24 00:27:12.398147 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:12.398060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d76449b94-4tmcr" event={"ID":"717409be-067d-4092-a201-0a1fe27a20ed","Type":"ContainerStarted","Data":"9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007"}
Jul 24 00:27:12.398147 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:12.398141    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d76449b94-4tmcr" event={"ID":"717409be-067d-4092-a201-0a1fe27a20ed","Type":"ContainerStarted","Data":"48ce4d6b882cf95d2420d08766f08b997429314e89d5d73c969002fd3a871f3a"}
Jul 24 00:27:12.398546 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:12.398176    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:12.417282 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:12.417230    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d76449b94-4tmcr" podStartSLOduration=1.8133306120000001 podStartE2EDuration="2.417214799s" totalImagesPullingTime="603.884187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:27:12.414659753 +0000 UTC m=+812.982475903" watchObservedRunningTime="2026-07-24 00:27:12.417214799 +0000 UTC m=+812.985030963"
Jul 24 00:27:23.405948 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:23.405913    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:28.055939 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.055899    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d76449b94-4tmcr"]
Jul 24 00:27:28.056386 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.056084    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d76449b94-4tmcr" podUID="717409be-067d-4092-a201-0a1fe27a20ed" containerName="manager" containerID="cri-o://9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007" gracePeriod=30
Jul 24 00:27:28.307039 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.306979    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:28.465948 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.465913    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/717409be-067d-4092-a201-0a1fe27a20ed-webhook-cert\") pod \"717409be-067d-4092-a201-0a1fe27a20ed\" (UID: \"717409be-067d-4092-a201-0a1fe27a20ed\") "
Jul 24 00:27:28.466128 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.465974    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qmh2\" (UniqueName: \"kubernetes.io/projected/717409be-067d-4092-a201-0a1fe27a20ed-kube-api-access-6qmh2\") pod \"717409be-067d-4092-a201-0a1fe27a20ed\" (UID: \"717409be-067d-4092-a201-0a1fe27a20ed\") "
Jul 24 00:27:28.468024 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.467981    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/717409be-067d-4092-a201-0a1fe27a20ed-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "717409be-067d-4092-a201-0a1fe27a20ed" (UID: "717409be-067d-4092-a201-0a1fe27a20ed"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:27:28.468140 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.468025    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717409be-067d-4092-a201-0a1fe27a20ed-kube-api-access-6qmh2" (OuterVolumeSpecName: "kube-api-access-6qmh2") pod "717409be-067d-4092-a201-0a1fe27a20ed" (UID: "717409be-067d-4092-a201-0a1fe27a20ed"). InnerVolumeSpecName "kube-api-access-6qmh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:27:28.475601 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.475570    2560 generic.go:356] "Generic (PLEG): container finished" podID="717409be-067d-4092-a201-0a1fe27a20ed" containerID="9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007" exitCode=0
Jul 24 00:27:28.475707 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.475617    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d76449b94-4tmcr"
Jul 24 00:27:28.475707 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.475615    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d76449b94-4tmcr" event={"ID":"717409be-067d-4092-a201-0a1fe27a20ed","Type":"ContainerDied","Data":"9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007"}
Jul 24 00:27:28.475788 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.475723    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d76449b94-4tmcr" event={"ID":"717409be-067d-4092-a201-0a1fe27a20ed","Type":"ContainerDied","Data":"48ce4d6b882cf95d2420d08766f08b997429314e89d5d73c969002fd3a871f3a"}
Jul 24 00:27:28.475788 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.475751    2560 scope.go:117] "RemoveContainer" containerID="9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007"
Jul 24 00:27:28.485460 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.485441    2560 scope.go:117] "RemoveContainer" containerID="9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007"
Jul 24 00:27:28.485741 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:27:28.485716    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007\": container with ID starting with 9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007 not found: ID does not exist" containerID="9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007"
Jul 24 00:27:28.485841 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.485745    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007"} err="failed to get container status \"9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007\": rpc error: code = NotFound desc = could not find container \"9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007\": container with ID starting with 9d9250082752f18880318ed4fa722f6910d70080837464bd6ce5b6f33e050007 not found: ID does not exist"
Jul 24 00:27:28.496056 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.496028    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d76449b94-4tmcr"]
Jul 24 00:27:28.499929 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.499901    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d76449b94-4tmcr"]
Jul 24 00:27:28.567149 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.567073    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/717409be-067d-4092-a201-0a1fe27a20ed-webhook-cert\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:27:28.567149 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:28.567099    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qmh2\" (UniqueName: \"kubernetes.io/projected/717409be-067d-4092-a201-0a1fe27a20ed-kube-api-access-6qmh2\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:27:29.320216 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.320181    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d76449b94-7w7cr"]
Jul 24 00:27:29.320641 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.320537    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="717409be-067d-4092-a201-0a1fe27a20ed" containerName="manager"
Jul 24 00:27:29.320641 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.320549    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="717409be-067d-4092-a201-0a1fe27a20ed" containerName="manager"
Jul 24 00:27:29.320641 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.320617    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="717409be-067d-4092-a201-0a1fe27a20ed" containerName="manager"
Jul 24 00:27:29.324924 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.324905    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d76449b94-7w7cr"
Jul 24 00:27:29.327999 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.327977    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 00:27:29.328099 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.327979    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-n6cg2\""
Jul 24 00:27:29.328099 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.327979    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 00:27:29.335273 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.335245    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d76449b94-7w7cr"]
Jul 24 00:27:29.474844 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.474805    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc25j\" (UniqueName: \"kubernetes.io/projected/9f943000-89ca-4640-8c6c-852915e4eb2d-kube-api-access-zc25j\") pod \"maas-controller-d76449b94-7w7cr\" (UID: \"9f943000-89ca-4640-8c6c-852915e4eb2d\") " pod="opendatahub/maas-controller-d76449b94-7w7cr"
Jul 24 00:27:29.475039 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.474930    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f943000-89ca-4640-8c6c-852915e4eb2d-webhook-cert\") pod \"maas-controller-d76449b94-7w7cr\" (UID: \"9f943000-89ca-4640-8c6c-852915e4eb2d\") " pod="opendatahub/maas-controller-d76449b94-7w7cr"
Jul 24 00:27:29.576350 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.576255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc25j\" (UniqueName: \"kubernetes.io/projected/9f943000-89ca-4640-8c6c-852915e4eb2d-kube-api-access-zc25j\") pod \"maas-controller-d76449b94-7w7cr\" (UID: \"9f943000-89ca-4640-8c6c-852915e4eb2d\") " pod="opendatahub/maas-controller-d76449b94-7w7cr"
Jul 24 00:27:29.576350 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.576333    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f943000-89ca-4640-8c6c-852915e4eb2d-webhook-cert\") pod \"maas-controller-d76449b94-7w7cr\" (UID: \"9f943000-89ca-4640-8c6c-852915e4eb2d\") " pod="opendatahub/maas-controller-d76449b94-7w7cr"
Jul 24 00:27:29.578774 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.578753    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f943000-89ca-4640-8c6c-852915e4eb2d-webhook-cert\") pod \"maas-controller-d76449b94-7w7cr\" (UID: \"9f943000-89ca-4640-8c6c-852915e4eb2d\") " pod="opendatahub/maas-controller-d76449b94-7w7cr"
Jul 24 00:27:29.584387 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.584365    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc25j\" (UniqueName: \"kubernetes.io/projected/9f943000-89ca-4640-8c6c-852915e4eb2d-kube-api-access-zc25j\") pod \"maas-controller-d76449b94-7w7cr\" (UID: \"9f943000-89ca-4640-8c6c-852915e4eb2d\") " pod="opendatahub/maas-controller-d76449b94-7w7cr"
Jul 24 00:27:29.636360 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.636330    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d76449b94-7w7cr"
Jul 24 00:27:29.766567 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.766542    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d76449b94-7w7cr"]
Jul 24 00:27:29.769295 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:27:29.769265    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f943000_89ca_4640_8c6c_852915e4eb2d.slice/crio-c7ae1415477a6e1a20af6c4871817cff8e98265440ad38f3089058032027ff76 WatchSource:0}: Error finding container c7ae1415477a6e1a20af6c4871817cff8e98265440ad38f3089058032027ff76: Status 404 returned error can't find the container with id c7ae1415477a6e1a20af6c4871817cff8e98265440ad38f3089058032027ff76
Jul 24 00:27:29.967520 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:29.967486    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="717409be-067d-4092-a201-0a1fe27a20ed" path="/var/lib/kubelet/pods/717409be-067d-4092-a201-0a1fe27a20ed/volumes"
Jul 24 00:27:30.487955 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:30.487854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d76449b94-7w7cr" event={"ID":"9f943000-89ca-4640-8c6c-852915e4eb2d","Type":"ContainerStarted","Data":"a4a7eab3a1164839cebb0457890bbd46ec24ef2f3062dae958a3cbee27ae2e72"}
Jul 24 00:27:30.488365 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:30.488030    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d76449b94-7w7cr"
Jul 24 00:27:30.488365 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:30.488055    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d76449b94-7w7cr" event={"ID":"9f943000-89ca-4640-8c6c-852915e4eb2d","Type":"ContainerStarted","Data":"c7ae1415477a6e1a20af6c4871817cff8e98265440ad38f3089058032027ff76"}
Jul 24 00:27:30.506121 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:30.506062    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d76449b94-7w7cr" podStartSLOduration=1.07546813 podStartE2EDuration="1.506042487s" totalImagesPullingTime="430.574357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:27:30.504902421 +0000 UTC m=+831.072718572" watchObservedRunningTime="2026-07-24 00:27:30.506042487 +0000 UTC m=+831.073858639"
Jul 24 00:27:41.495344 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:27:41.495313    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d76449b94-7w7cr"
Jul 24 00:36:05.002359 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.002265    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"]
Jul 24 00:36:05.005276 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.005251    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.007782 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.007720    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3cc92572-kserve-self-signed-certs\""
Jul 24 00:36:05.021315 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.021292    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"]
Jul 24 00:36:05.121257 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.121206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnvfw\" (UniqueName: \"kubernetes.io/projected/c21eb5af-612e-4ce7-9d42-45fb261c66d9-kube-api-access-tnvfw\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.121257 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.121261    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-model-cache\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.121518 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.121345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tls-certs\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.121518 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.121413    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tmp-dir\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.121518 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.121456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-dshm\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.121616 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.121545    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-home\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.222688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.222652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tls-certs\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.222921 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.222707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tmp-dir\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.222921 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.222734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-dshm\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.222921 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.222767    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-home\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.222921 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.222789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnvfw\" (UniqueName: \"kubernetes.io/projected/c21eb5af-612e-4ce7-9d42-45fb261c66d9-kube-api-access-tnvfw\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.222921 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.222811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-model-cache\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.223209 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.223184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tmp-dir\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.223256 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.223206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-model-cache\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.223256 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.223246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-home\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.225110 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.225084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-dshm\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.225465 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.225442    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tls-certs\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.230944 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.230923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnvfw\" (UniqueName: \"kubernetes.io/projected/c21eb5af-612e-4ce7-9d42-45fb261c66d9-kube-api-access-tnvfw\") pod \"e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") " pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.314796 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.314713    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:05.456284 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.456249    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"]
Jul 24 00:36:05.459621 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:36:05.459595    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc21eb5af_612e_4ce7_9d42_45fb261c66d9.slice/crio-d3d5844048dceaa511d7512213280e9a28a5cf6f55b5f9404f2f4518fb761d42 WatchSource:0}: Error finding container d3d5844048dceaa511d7512213280e9a28a5cf6f55b5f9404f2f4518fb761d42: Status 404 returned error can't find the container with id d3d5844048dceaa511d7512213280e9a28a5cf6f55b5f9404f2f4518fb761d42
Jul 24 00:36:05.801976 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.801942    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6" event={"ID":"c21eb5af-612e-4ce7-9d42-45fb261c66d9","Type":"ContainerStarted","Data":"882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61"}
Jul 24 00:36:05.801976 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.801981    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6" event={"ID":"c21eb5af-612e-4ce7-9d42-45fb261c66d9","Type":"ContainerStarted","Data":"d3d5844048dceaa511d7512213280e9a28a5cf6f55b5f9404f2f4518fb761d42"}
Jul 24 00:36:05.820259 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:05.820206    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6" podStartSLOduration=1.820187682 podStartE2EDuration="1.820187682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:36:05.818159112 +0000 UTC m=+1346.385975262" watchObservedRunningTime="2026-07-24 00:36:05.820187682 +0000 UTC m=+1346.388003831"
Jul 24 00:36:15.315648 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:15.315609    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:15.316125 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:15.315666    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:15.328243 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:15.328211    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:15.863188 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:15.863151    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:31.357559 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:31.357522    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"]
Jul 24 00:36:46.378442 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.378384    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6" podUID="c21eb5af-612e-4ce7-9d42-45fb261c66d9" containerName="main" containerID="cri-o://882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61" gracePeriod=45
Jul 24 00:36:46.634939 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.634876    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:46.686383 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.686350    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-dshm\") pod \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") "
Jul 24 00:36:46.686551 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.686395    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tls-certs\") pod \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") "
Jul 24 00:36:46.686551 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.686452    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-home\") pod \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") "
Jul 24 00:36:46.686551 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.686481    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tmp-dir\") pod \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") "
Jul 24 00:36:46.686551 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.686507    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-model-cache\") pod \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") "
Jul 24 00:36:46.686773 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.686647    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnvfw\" (UniqueName: \"kubernetes.io/projected/c21eb5af-612e-4ce7-9d42-45fb261c66d9-kube-api-access-tnvfw\") pod \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\" (UID: \"c21eb5af-612e-4ce7-9d42-45fb261c66d9\") "
Jul 24 00:36:46.686773 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.686748    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c21eb5af-612e-4ce7-9d42-45fb261c66d9" (UID: "c21eb5af-612e-4ce7-9d42-45fb261c66d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:36:46.686895 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.686762    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c21eb5af-612e-4ce7-9d42-45fb261c66d9" (UID: "c21eb5af-612e-4ce7-9d42-45fb261c66d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:36:46.686895 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.686772    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-home" (OuterVolumeSpecName: "home") pod "c21eb5af-612e-4ce7-9d42-45fb261c66d9" (UID: "c21eb5af-612e-4ce7-9d42-45fb261c66d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:36:46.687022 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.687000    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tmp-dir\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:36:46.687058 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.687029    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-model-cache\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:36:46.687058 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.687043    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-home\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:36:46.688775 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.688749    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-dshm" (OuterVolumeSpecName: "dshm") pod "c21eb5af-612e-4ce7-9d42-45fb261c66d9" (UID: "c21eb5af-612e-4ce7-9d42-45fb261c66d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:36:46.688898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.688755    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21eb5af-612e-4ce7-9d42-45fb261c66d9-kube-api-access-tnvfw" (OuterVolumeSpecName: "kube-api-access-tnvfw") pod "c21eb5af-612e-4ce7-9d42-45fb261c66d9" (UID: "c21eb5af-612e-4ce7-9d42-45fb261c66d9"). InnerVolumeSpecName "kube-api-access-tnvfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:36:46.688898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.688777    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c21eb5af-612e-4ce7-9d42-45fb261c66d9" (UID: "c21eb5af-612e-4ce7-9d42-45fb261c66d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:36:46.787820 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.787786    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnvfw\" (UniqueName: \"kubernetes.io/projected/c21eb5af-612e-4ce7-9d42-45fb261c66d9-kube-api-access-tnvfw\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:36:46.787820 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.787816    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c21eb5af-612e-4ce7-9d42-45fb261c66d9-dshm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:36:46.787820 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.787859    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c21eb5af-612e-4ce7-9d42-45fb261c66d9-tls-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:36:46.996475 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.996443    2560 generic.go:356] "Generic (PLEG): container finished" podID="c21eb5af-612e-4ce7-9d42-45fb261c66d9" containerID="882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61" exitCode=0
Jul 24 00:36:46.996643 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.996483    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"
Jul 24 00:36:46.996643 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.996528    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6" event={"ID":"c21eb5af-612e-4ce7-9d42-45fb261c66d9","Type":"ContainerDied","Data":"882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61"}
Jul 24 00:36:46.996643 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.996577    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6" event={"ID":"c21eb5af-612e-4ce7-9d42-45fb261c66d9","Type":"ContainerDied","Data":"d3d5844048dceaa511d7512213280e9a28a5cf6f55b5f9404f2f4518fb761d42"}
Jul 24 00:36:46.996643 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:46.996601    2560 scope.go:117] "RemoveContainer" containerID="882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61"
Jul 24 00:36:47.005753 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:47.005733    2560 scope.go:117] "RemoveContainer" containerID="882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61"
Jul 24 00:36:47.006108 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:36:47.006089    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61\": container with ID starting with 882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61 not found: ID does not exist" containerID="882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61"
Jul 24 00:36:47.006173 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:47.006117    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61"} err="failed to get container status \"882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61\": rpc error: code = NotFound desc = could not find container \"882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61\": container with ID starting with 882d7db6c87e3e825a420808ecf7814c0e7d4f3b8d432b1b06feafc871e3ce61 not found: ID does not exist"
Jul 24 00:36:47.027314 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:47.027289    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"]
Jul 24 00:36:47.038441 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:47.038411    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3cc92572-kserve-5d9468f4ff-hhnm6"]
Jul 24 00:36:47.967131 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:36:47.967095    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c21eb5af-612e-4ce7-9d42-45fb261c66d9" path="/var/lib/kubelet/pods/c21eb5af-612e-4ce7-9d42-45fb261c66d9/volumes"
Jul 24 00:40:18.601876 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.601813    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"]
Jul 24 00:40:18.602335 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.602184    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21eb5af-612e-4ce7-9d42-45fb261c66d9" containerName="main"
Jul 24 00:40:18.602335 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.602196    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21eb5af-612e-4ce7-9d42-45fb261c66d9" containerName="main"
Jul 24 00:40:18.602335 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.602266    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c21eb5af-612e-4ce7-9d42-45fb261c66d9" containerName="main"
Jul 24 00:40:18.605240 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.605220    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:18.607584 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.607556    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-edc8c8a3-serving-cert\""
Jul 24 00:40:18.616277 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.616253    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"]
Jul 24 00:40:18.711886 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.711822    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1b3567-fa49-47d8-b29b-d307c2907ed4-maas-api-tls\") pod \"maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:18.712050 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.711912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b3567-fa49-47d8-b29b-d307c2907ed4-openshift-service-ca\") pod \"maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:18.712050 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.711973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmpvw\" (UniqueName: \"kubernetes.io/projected/fe1b3567-fa49-47d8-b29b-d307c2907ed4-kube-api-access-tmpvw\") pod \"maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:18.813252 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.813214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1b3567-fa49-47d8-b29b-d307c2907ed4-maas-api-tls\") pod \"maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:18.813427 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.813261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b3567-fa49-47d8-b29b-d307c2907ed4-openshift-service-ca\") pod \"maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:18.813427 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.813296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmpvw\" (UniqueName: \"kubernetes.io/projected/fe1b3567-fa49-47d8-b29b-d307c2907ed4-kube-api-access-tmpvw\") pod \"maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:18.813427 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:40:18.813352    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-serving-cert: secret "maas-api-e2e-ait-edc8c8a3-serving-cert" not found
Jul 24 00:40:18.813549 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:40:18.813450    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe1b3567-fa49-47d8-b29b-d307c2907ed4-maas-api-tls podName:fe1b3567-fa49-47d8-b29b-d307c2907ed4 nodeName:}" failed. No retries permitted until 2026-07-24 00:40:19.31342592 +0000 UTC m=+1599.881242054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe1b3567-fa49-47d8-b29b-d307c2907ed4-maas-api-tls") pod "maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww" (UID: "fe1b3567-fa49-47d8-b29b-d307c2907ed4") : secret "maas-api-e2e-ait-edc8c8a3-serving-cert" not found
Jul 24 00:40:18.814120 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.814095    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b3567-fa49-47d8-b29b-d307c2907ed4-openshift-service-ca\") pod \"maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:18.821987 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:18.821965    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmpvw\" (UniqueName: \"kubernetes.io/projected/fe1b3567-fa49-47d8-b29b-d307c2907ed4-kube-api-access-tmpvw\") pod \"maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:19.318251 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.318209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1b3567-fa49-47d8-b29b-d307c2907ed4-maas-api-tls\") pod \"maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:19.320799 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.320761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1b3567-fa49-47d8-b29b-d307c2907ed4-maas-api-tls\") pod \"maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:19.340641 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.340602    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"]
Jul 24 00:40:19.344285 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.344263    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:19.346692 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.346668    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-edc8c8a3-dockercfg-rmd4m\""
Jul 24 00:40:19.346813 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.346674    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-edc8c8a3\""
Jul 24 00:40:19.352971 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.352947    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"]
Jul 24 00:40:19.379497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.379459    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"]
Jul 24 00:40:19.382708 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.382692    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:19.398480 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.398453    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"]
Jul 24 00:40:19.419190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.419158    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598948ff-422a-4c9a-8b54-a894558a94c0-plugins-config-volume\") pod \"payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw\" (UID: \"598948ff-422a-4c9a-8b54-a894558a94c0\") " pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:19.419321 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.419261    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52b8j\" (UniqueName: \"kubernetes.io/projected/598948ff-422a-4c9a-8b54-a894558a94c0-kube-api-access-52b8j\") pod \"payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw\" (UID: \"598948ff-422a-4c9a-8b54-a894558a94c0\") " pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:19.516407 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.516373    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:19.520488 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.520470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52b8j\" (UniqueName: \"kubernetes.io/projected/598948ff-422a-4c9a-8b54-a894558a94c0-kube-api-access-52b8j\") pod \"payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw\" (UID: \"598948ff-422a-4c9a-8b54-a894558a94c0\") " pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:19.520548 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.520505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9j95\" (UniqueName: \"kubernetes.io/projected/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-kube-api-access-w9j95\") pod \"payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn\" (UID: \"61cde3c0-37e3-46f0-a1d9-26be780f8c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:19.520548 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.520539    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn\" (UID: \"61cde3c0-37e3-46f0-a1d9-26be780f8c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:19.520655 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.520559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598948ff-422a-4c9a-8b54-a894558a94c0-plugins-config-volume\") pod \"payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw\" (UID: \"598948ff-422a-4c9a-8b54-a894558a94c0\") " pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:19.521426 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.521399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598948ff-422a-4c9a-8b54-a894558a94c0-plugins-config-volume\") pod \"payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw\" (UID: \"598948ff-422a-4c9a-8b54-a894558a94c0\") " pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:19.536887 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.536851    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52b8j\" (UniqueName: \"kubernetes.io/projected/598948ff-422a-4c9a-8b54-a894558a94c0-kube-api-access-52b8j\") pod \"payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw\" (UID: \"598948ff-422a-4c9a-8b54-a894558a94c0\") " pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:19.621490 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.621451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9j95\" (UniqueName: \"kubernetes.io/projected/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-kube-api-access-w9j95\") pod \"payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn\" (UID: \"61cde3c0-37e3-46f0-a1d9-26be780f8c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:19.621944 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.621529    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn\" (UID: \"61cde3c0-37e3-46f0-a1d9-26be780f8c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:19.622413 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.622376    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn\" (UID: \"61cde3c0-37e3-46f0-a1d9-26be780f8c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:19.630769 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.630742    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9j95\" (UniqueName: \"kubernetes.io/projected/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-kube-api-access-w9j95\") pod \"payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn\" (UID: \"61cde3c0-37e3-46f0-a1d9-26be780f8c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:19.656583 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.656546    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:19.670748 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.670722    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"]
Jul 24 00:40:19.673694 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:40:19.673665    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1b3567_fa49_47d8_b29b_d307c2907ed4.slice/crio-dc0de7294ca22c214d5e66999b549355e58c5465a44f0e068edb0e474ba7c65d WatchSource:0}: Error finding container dc0de7294ca22c214d5e66999b549355e58c5465a44f0e068edb0e474ba7c65d: Status 404 returned error can't find the container with id dc0de7294ca22c214d5e66999b549355e58c5465a44f0e068edb0e474ba7c65d
Jul 24 00:40:19.674994 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.674974    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:40:19.692377 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.692349    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:19.839753 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.839665    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"]
Jul 24 00:40:19.842121 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:40:19.842093    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598948ff_422a_4c9a_8b54_a894558a94c0.slice/crio-468de1f46d148e8f3bbbc9a97583bf4cfe5ea9080f3af2b522a462c583e36af7 WatchSource:0}: Error finding container 468de1f46d148e8f3bbbc9a97583bf4cfe5ea9080f3af2b522a462c583e36af7: Status 404 returned error can't find the container with id 468de1f46d148e8f3bbbc9a97583bf4cfe5ea9080f3af2b522a462c583e36af7
Jul 24 00:40:19.869149 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.869120    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"]
Jul 24 00:40:19.877111 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:40:19.877084    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cde3c0_37e3_46f0_a1d9_26be780f8c7b.slice/crio-96d38dbac942fa64300e1435e7975c015434d48602c222ec43587a8d80df667d WatchSource:0}: Error finding container 96d38dbac942fa64300e1435e7975c015434d48602c222ec43587a8d80df667d: Status 404 returned error can't find the container with id 96d38dbac942fa64300e1435e7975c015434d48602c222ec43587a8d80df667d
Jul 24 00:40:19.979415 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.979382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw" event={"ID":"598948ff-422a-4c9a-8b54-a894558a94c0","Type":"ContainerStarted","Data":"7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d"}
Jul 24 00:40:19.979593 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.979437    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw" event={"ID":"598948ff-422a-4c9a-8b54-a894558a94c0","Type":"ContainerStarted","Data":"468de1f46d148e8f3bbbc9a97583bf4cfe5ea9080f3af2b522a462c583e36af7"}
Jul 24 00:40:19.979593 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.979484    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:19.980670 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.980643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww" event={"ID":"fe1b3567-fa49-47d8-b29b-d307c2907ed4","Type":"ContainerStarted","Data":"dc0de7294ca22c214d5e66999b549355e58c5465a44f0e068edb0e474ba7c65d"}
Jul 24 00:40:19.982070 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.982043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn" event={"ID":"61cde3c0-37e3-46f0-a1d9-26be780f8c7b","Type":"ContainerStarted","Data":"743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89"}
Jul 24 00:40:19.982189 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.982084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn" event={"ID":"61cde3c0-37e3-46f0-a1d9-26be780f8c7b","Type":"ContainerStarted","Data":"96d38dbac942fa64300e1435e7975c015434d48602c222ec43587a8d80df667d"}
Jul 24 00:40:19.982189 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:19.982133    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:20.016563 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:20.016499    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw" podStartSLOduration=1.016481541 podStartE2EDuration="1.016481541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:40:20.01516946 +0000 UTC m=+1600.582985613" watchObservedRunningTime="2026-07-24 00:40:20.016481541 +0000 UTC m=+1600.584297690"
Jul 24 00:40:20.044874 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:20.044790    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn" podStartSLOduration=1.044768795 podStartE2EDuration="1.044768795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:40:20.04297314 +0000 UTC m=+1600.610789292" watchObservedRunningTime="2026-07-24 00:40:20.044768795 +0000 UTC m=+1600.612584949"
Jul 24 00:40:20.988234 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:20.988194    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww" event={"ID":"fe1b3567-fa49-47d8-b29b-d307c2907ed4","Type":"ContainerStarted","Data":"f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553"}
Jul 24 00:40:20.988675 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:20.988402    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:21.009488 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:21.009434    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww" podStartSLOduration=2.491406886 podStartE2EDuration="3.009418924s" totalImagesPullingTime="518.012038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:40:21.008439962 +0000 UTC m=+1601.576256125" watchObservedRunningTime="2026-07-24 00:40:21.009418924 +0000 UTC m=+1601.577235073"
Jul 24 00:40:26.999271 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:26.999240    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:30.989971 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:30.989941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:30.990343 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:30.990009    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:54.504265 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.504175    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"]
Jul 24 00:40:54.504819 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.504428    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww" podUID="fe1b3567-fa49-47d8-b29b-d307c2907ed4" containerName="maas-api" containerID="cri-o://f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553" gracePeriod=30
Jul 24 00:40:54.561646 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.561603    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"]
Jul 24 00:40:54.562346 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.562299    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw" podUID="598948ff-422a-4c9a-8b54-a894558a94c0" containerName="payload-processing" containerID="cri-o://7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d" gracePeriod=30
Jul 24 00:40:54.570869 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.570801    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"]
Jul 24 00:40:54.571306 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.571255    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn" podUID="61cde3c0-37e3-46f0-a1d9-26be780f8c7b" containerName="payload-pre-processing" containerID="cri-o://743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89" gracePeriod=30
Jul 24 00:40:54.832455 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.832426    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:54.838726 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.838698    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:54.876277 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.876246    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:54.920666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.920630    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmpvw\" (UniqueName: \"kubernetes.io/projected/fe1b3567-fa49-47d8-b29b-d307c2907ed4-kube-api-access-tmpvw\") pod \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") "
Jul 24 00:40:54.920886 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.920694    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b3567-fa49-47d8-b29b-d307c2907ed4-openshift-service-ca\") pod \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") "
Jul 24 00:40:54.920886 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.920713    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1b3567-fa49-47d8-b29b-d307c2907ed4-maas-api-tls\") pod \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\" (UID: \"fe1b3567-fa49-47d8-b29b-d307c2907ed4\") "
Jul 24 00:40:54.920886 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.920740    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52b8j\" (UniqueName: \"kubernetes.io/projected/598948ff-422a-4c9a-8b54-a894558a94c0-kube-api-access-52b8j\") pod \"598948ff-422a-4c9a-8b54-a894558a94c0\" (UID: \"598948ff-422a-4c9a-8b54-a894558a94c0\") "
Jul 24 00:40:54.920886 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.920776    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-plugins-config-volume\") pod \"61cde3c0-37e3-46f0-a1d9-26be780f8c7b\" (UID: \"61cde3c0-37e3-46f0-a1d9-26be780f8c7b\") "
Jul 24 00:40:54.920886 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.920800    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598948ff-422a-4c9a-8b54-a894558a94c0-plugins-config-volume\") pod \"598948ff-422a-4c9a-8b54-a894558a94c0\" (UID: \"598948ff-422a-4c9a-8b54-a894558a94c0\") "
Jul 24 00:40:54.920886 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.920820    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9j95\" (UniqueName: \"kubernetes.io/projected/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-kube-api-access-w9j95\") pod \"61cde3c0-37e3-46f0-a1d9-26be780f8c7b\" (UID: \"61cde3c0-37e3-46f0-a1d9-26be780f8c7b\") "
Jul 24 00:40:54.921253 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.921219    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598948ff-422a-4c9a-8b54-a894558a94c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "598948ff-422a-4c9a-8b54-a894558a94c0" (UID: "598948ff-422a-4c9a-8b54-a894558a94c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:40:54.921253 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.921237    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "61cde3c0-37e3-46f0-a1d9-26be780f8c7b" (UID: "61cde3c0-37e3-46f0-a1d9-26be780f8c7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:40:54.921511 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.921483    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1b3567-fa49-47d8-b29b-d307c2907ed4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe1b3567-fa49-47d8-b29b-d307c2907ed4" (UID: "fe1b3567-fa49-47d8-b29b-d307c2907ed4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:40:54.923092 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.923069    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1b3567-fa49-47d8-b29b-d307c2907ed4-kube-api-access-tmpvw" (OuterVolumeSpecName: "kube-api-access-tmpvw") pod "fe1b3567-fa49-47d8-b29b-d307c2907ed4" (UID: "fe1b3567-fa49-47d8-b29b-d307c2907ed4"). InnerVolumeSpecName "kube-api-access-tmpvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:40:54.923186 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.923102    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-kube-api-access-w9j95" (OuterVolumeSpecName: "kube-api-access-w9j95") pod "61cde3c0-37e3-46f0-a1d9-26be780f8c7b" (UID: "61cde3c0-37e3-46f0-a1d9-26be780f8c7b"). InnerVolumeSpecName "kube-api-access-w9j95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:40:54.923374 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.923346    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1b3567-fa49-47d8-b29b-d307c2907ed4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe1b3567-fa49-47d8-b29b-d307c2907ed4" (UID: "fe1b3567-fa49-47d8-b29b-d307c2907ed4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:40:54.923455 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:54.923352    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598948ff-422a-4c9a-8b54-a894558a94c0-kube-api-access-52b8j" (OuterVolumeSpecName: "kube-api-access-52b8j") pod "598948ff-422a-4c9a-8b54-a894558a94c0" (UID: "598948ff-422a-4c9a-8b54-a894558a94c0"). InnerVolumeSpecName "kube-api-access-52b8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:40:55.021728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.021641    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52b8j\" (UniqueName: \"kubernetes.io/projected/598948ff-422a-4c9a-8b54-a894558a94c0-kube-api-access-52b8j\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:40:55.021728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.021673    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:40:55.021728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.021682    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598948ff-422a-4c9a-8b54-a894558a94c0-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:40:55.021728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.021691    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9j95\" (UniqueName: \"kubernetes.io/projected/61cde3c0-37e3-46f0-a1d9-26be780f8c7b-kube-api-access-w9j95\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:40:55.021728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.021703    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmpvw\" (UniqueName: \"kubernetes.io/projected/fe1b3567-fa49-47d8-b29b-d307c2907ed4-kube-api-access-tmpvw\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:40:55.021728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.021711    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b3567-fa49-47d8-b29b-d307c2907ed4-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:40:55.021728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.021722    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1b3567-fa49-47d8-b29b-d307c2907ed4-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:40:55.155915 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.155879    2560 generic.go:356] "Generic (PLEG): container finished" podID="61cde3c0-37e3-46f0-a1d9-26be780f8c7b" containerID="743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89" exitCode=0
Jul 24 00:40:55.156090 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.155930    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"
Jul 24 00:40:55.156090 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.155967    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn" event={"ID":"61cde3c0-37e3-46f0-a1d9-26be780f8c7b","Type":"ContainerDied","Data":"743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89"}
Jul 24 00:40:55.156090 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.156024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn" event={"ID":"61cde3c0-37e3-46f0-a1d9-26be780f8c7b","Type":"ContainerDied","Data":"96d38dbac942fa64300e1435e7975c015434d48602c222ec43587a8d80df667d"}
Jul 24 00:40:55.156090 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.156056    2560 scope.go:117] "RemoveContainer" containerID="743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89"
Jul 24 00:40:55.157448 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.157421    2560 generic.go:356] "Generic (PLEG): container finished" podID="598948ff-422a-4c9a-8b54-a894558a94c0" containerID="7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d" exitCode=0
Jul 24 00:40:55.157559 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.157462    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"
Jul 24 00:40:55.157559 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.157509    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw" event={"ID":"598948ff-422a-4c9a-8b54-a894558a94c0","Type":"ContainerDied","Data":"7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d"}
Jul 24 00:40:55.157559 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.157552    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw" event={"ID":"598948ff-422a-4c9a-8b54-a894558a94c0","Type":"ContainerDied","Data":"468de1f46d148e8f3bbbc9a97583bf4cfe5ea9080f3af2b522a462c583e36af7"}
Jul 24 00:40:55.159019 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.158993    2560 generic.go:356] "Generic (PLEG): container finished" podID="fe1b3567-fa49-47d8-b29b-d307c2907ed4" containerID="f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553" exitCode=0
Jul 24 00:40:55.159137 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.159041    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"
Jul 24 00:40:55.159137 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.159047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww" event={"ID":"fe1b3567-fa49-47d8-b29b-d307c2907ed4","Type":"ContainerDied","Data":"f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553"}
Jul 24 00:40:55.159251 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.159170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww" event={"ID":"fe1b3567-fa49-47d8-b29b-d307c2907ed4","Type":"ContainerDied","Data":"dc0de7294ca22c214d5e66999b549355e58c5465a44f0e068edb0e474ba7c65d"}
Jul 24 00:40:55.166423 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.166399    2560 scope.go:117] "RemoveContainer" containerID="743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89"
Jul 24 00:40:55.166764 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:40:55.166728    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89\": container with ID starting with 743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89 not found: ID does not exist" containerID="743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89"
Jul 24 00:40:55.166876 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.166765    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89"} err="failed to get container status \"743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89\": rpc error: code = NotFound desc = could not find container \"743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89\": container with ID starting with 743c9a2357672f620a03a60c985122f1a307187ae02684412c55720f586d9f89 not found: ID does not exist"
Jul 24 00:40:55.166876 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.166809    2560 scope.go:117] "RemoveContainer" containerID="7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d"
Jul 24 00:40:55.176743 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.176722    2560 scope.go:117] "RemoveContainer" containerID="7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d"
Jul 24 00:40:55.177106 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:40:55.177087    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d\": container with ID starting with 7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d not found: ID does not exist" containerID="7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d"
Jul 24 00:40:55.177177 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.177112    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d"} err="failed to get container status \"7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d\": rpc error: code = NotFound desc = could not find container \"7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d\": container with ID starting with 7aee8240cb4f6897244787de42897e02baf62c375da37569ab277d2dc5fbf82d not found: ID does not exist"
Jul 24 00:40:55.177177 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.177131    2560 scope.go:117] "RemoveContainer" containerID="f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553"
Jul 24 00:40:55.183599 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.183573    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"]
Jul 24 00:40:55.186559 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.186532    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc8c8a3-57cc6db95b-2kdvn"]
Jul 24 00:40:55.186995 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.186971    2560 scope.go:117] "RemoveContainer" containerID="f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553"
Jul 24 00:40:55.187264 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:40:55.187245    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553\": container with ID starting with f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553 not found: ID does not exist" containerID="f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553"
Jul 24 00:40:55.187327 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.187270    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553"} err="failed to get container status \"f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553\": rpc error: code = NotFound desc = could not find container \"f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553\": container with ID starting with f06354bccde056c3d226ad5004bd82e9d61a48fe2f866d98607bf5e4bb60c553 not found: ID does not exist"
Jul 24 00:40:55.195195 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.195174    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"]
Jul 24 00:40:55.198814 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.198793    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc8c8a3-5f55ccccb6-h69fw"]
Jul 24 00:40:55.208025 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.208006    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"]
Jul 24 00:40:55.211977 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.211956    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc8c8a3-555cc89d95-fnwww"]
Jul 24 00:40:55.967338 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.967307    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="598948ff-422a-4c9a-8b54-a894558a94c0" path="/var/lib/kubelet/pods/598948ff-422a-4c9a-8b54-a894558a94c0/volumes"
Jul 24 00:40:55.967701 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.967639    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61cde3c0-37e3-46f0-a1d9-26be780f8c7b" path="/var/lib/kubelet/pods/61cde3c0-37e3-46f0-a1d9-26be780f8c7b/volumes"
Jul 24 00:40:55.968006 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:40:55.967993    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe1b3567-fa49-47d8-b29b-d307c2907ed4" path="/var/lib/kubelet/pods/fe1b3567-fa49-47d8-b29b-d307c2907ed4/volumes"
Jul 24 00:41:03.003489 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.003451    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"]
Jul 24 00:41:03.005977 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.004004    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598948ff-422a-4c9a-8b54-a894558a94c0" containerName="payload-processing"
Jul 24 00:41:03.005977 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.004021    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="598948ff-422a-4c9a-8b54-a894558a94c0" containerName="payload-processing"
Jul 24 00:41:03.005977 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.004033    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe1b3567-fa49-47d8-b29b-d307c2907ed4" containerName="maas-api"
Jul 24 00:41:03.005977 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.004042    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe1b3567-fa49-47d8-b29b-d307c2907ed4" containerName="maas-api"
Jul 24 00:41:03.005977 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.004068    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61cde3c0-37e3-46f0-a1d9-26be780f8c7b" containerName="payload-pre-processing"
Jul 24 00:41:03.005977 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.004077    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="61cde3c0-37e3-46f0-a1d9-26be780f8c7b" containerName="payload-pre-processing"
Jul 24 00:41:03.005977 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.004168    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="598948ff-422a-4c9a-8b54-a894558a94c0" containerName="payload-processing"
Jul 24 00:41:03.005977 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.004184    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe1b3567-fa49-47d8-b29b-d307c2907ed4" containerName="maas-api"
Jul 24 00:41:03.005977 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.004193    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="61cde3c0-37e3-46f0-a1d9-26be780f8c7b" containerName="payload-pre-processing"
Jul 24 00:41:03.007345 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.007324    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.009787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.009762    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-eaba160e-serving-cert\""
Jul 24 00:41:03.019205 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.019177    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"]
Jul 24 00:41:03.094780 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.094745    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-openshift-service-ca\") pod \"maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.094961 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.094793    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-maas-api-tls\") pod \"maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.094961 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.094870    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhlbr\" (UniqueName: \"kubernetes.io/projected/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-kube-api-access-dhlbr\") pod \"maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.196272 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.196237    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-openshift-service-ca\") pod \"maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.196470 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.196295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-maas-api-tls\") pod \"maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.196470 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.196333    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhlbr\" (UniqueName: \"kubernetes.io/projected/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-kube-api-access-dhlbr\") pod \"maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.196470 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:41:03.196463    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-serving-cert: secret "maas-api-e2e-migrate-eaba160e-serving-cert" not found
Jul 24 00:41:03.196657 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:41:03.196542    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-maas-api-tls podName:d8985c15-4674-4b6c-a42a-b9f77d2b57a0 nodeName:}" failed. No retries permitted until 2026-07-24 00:41:03.696519331 +0000 UTC m=+1644.264335464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-maas-api-tls") pod "maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv" (UID: "d8985c15-4674-4b6c-a42a-b9f77d2b57a0") : secret "maas-api-e2e-migrate-eaba160e-serving-cert" not found
Jul 24 00:41:03.197038 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.197016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-openshift-service-ca\") pod \"maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.204218 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.204196    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhlbr\" (UniqueName: \"kubernetes.io/projected/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-kube-api-access-dhlbr\") pod \"maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.588350 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.588312    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"]
Jul 24 00:41:03.591887 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.591870    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:03.594460 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.594437    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-eaba160e-dockercfg-j2p7g\""
Jul 24 00:41:03.594645 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.594631    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-eaba160e\""
Jul 24 00:41:03.608527 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.608491    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"]
Jul 24 00:41:03.616845 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.616796    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"]
Jul 24 00:41:03.620066 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.620043    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:03.631637 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.631607    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"]
Jul 24 00:41:03.702050 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.702013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-plugins-config-volume\") pod \"payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4\" (UID: \"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568\") " pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:03.702243 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.702099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-maas-api-tls\") pod \"maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.702243 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.702138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt\" (UID: \"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:03.702243 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.702164    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mmg4\" (UniqueName: \"kubernetes.io/projected/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-kube-api-access-8mmg4\") pod \"payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt\" (UID: \"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:03.702243 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.702203    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ndv\" (UniqueName: \"kubernetes.io/projected/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-kube-api-access-h7ndv\") pod \"payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4\" (UID: \"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568\") " pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:03.704710 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.704684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-maas-api-tls\") pod \"maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.803219 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.803163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt\" (UID: \"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:03.803219 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.803219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mmg4\" (UniqueName: \"kubernetes.io/projected/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-kube-api-access-8mmg4\") pod \"payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt\" (UID: \"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:03.803502 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.803259    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ndv\" (UniqueName: \"kubernetes.io/projected/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-kube-api-access-h7ndv\") pod \"payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4\" (UID: \"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568\") " pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:03.803502 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.803427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-plugins-config-volume\") pod \"payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4\" (UID: \"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568\") " pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:03.804154 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.804130    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt\" (UID: \"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:03.804270 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.804254    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-plugins-config-volume\") pod \"payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4\" (UID: \"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568\") " pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:03.811631 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.811603    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mmg4\" (UniqueName: \"kubernetes.io/projected/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-kube-api-access-8mmg4\") pod \"payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt\" (UID: \"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:03.811780 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.811635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ndv\" (UniqueName: \"kubernetes.io/projected/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-kube-api-access-h7ndv\") pod \"payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4\" (UID: \"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568\") " pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:03.903218 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.903134    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:03.918233 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.918201    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:03.931498 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:03.931468    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:04.062683 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:04.061734    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"]
Jul 24 00:41:04.067977 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:41:04.067946    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1844e56_5be6_46ea_9e56_1cf23cd9d6b8.slice/crio-2d07424e8f3bcbb7806c11f4d36205d06a5a4844eb3582c094d28994d59be107 WatchSource:0}: Error finding container 2d07424e8f3bcbb7806c11f4d36205d06a5a4844eb3582c094d28994d59be107: Status 404 returned error can't find the container with id 2d07424e8f3bcbb7806c11f4d36205d06a5a4844eb3582c094d28994d59be107
Jul 24 00:41:04.095381 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:04.095300    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"]
Jul 24 00:41:04.102113 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:41:04.102081    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8985c15_4674_4b6c_a42a_b9f77d2b57a0.slice/crio-fe805fbf169c48b7dee2f2f55a8d7558c17b4bf22e9cf7016fceddd23a1df586 WatchSource:0}: Error finding container fe805fbf169c48b7dee2f2f55a8d7558c17b4bf22e9cf7016fceddd23a1df586: Status 404 returned error can't find the container with id fe805fbf169c48b7dee2f2f55a8d7558c17b4bf22e9cf7016fceddd23a1df586
Jul 24 00:41:04.118320 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:04.118292    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"]
Jul 24 00:41:04.121032 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:41:04.121006    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ec5c73_8f7e_485c_b6e6_ab6f9bdcf568.slice/crio-ce219d7e3097673a99cef53c4057f2819877cf9e723dcea2f3e911af33b04e2a WatchSource:0}: Error finding container ce219d7e3097673a99cef53c4057f2819877cf9e723dcea2f3e911af33b04e2a: Status 404 returned error can't find the container with id ce219d7e3097673a99cef53c4057f2819877cf9e723dcea2f3e911af33b04e2a
Jul 24 00:41:04.206213 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:04.206175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv" event={"ID":"d8985c15-4674-4b6c-a42a-b9f77d2b57a0","Type":"ContainerStarted","Data":"fe805fbf169c48b7dee2f2f55a8d7558c17b4bf22e9cf7016fceddd23a1df586"}
Jul 24 00:41:04.207988 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:04.207962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4" event={"ID":"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568","Type":"ContainerStarted","Data":"b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549"}
Jul 24 00:41:04.208116 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:04.208011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4" event={"ID":"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568","Type":"ContainerStarted","Data":"ce219d7e3097673a99cef53c4057f2819877cf9e723dcea2f3e911af33b04e2a"}
Jul 24 00:41:04.208116 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:04.208087    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:04.210017 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:04.209993    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt" event={"ID":"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8","Type":"ContainerStarted","Data":"c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579"}
Jul 24 00:41:04.210094 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:04.210041    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt" event={"ID":"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8","Type":"ContainerStarted","Data":"2d07424e8f3bcbb7806c11f4d36205d06a5a4844eb3582c094d28994d59be107"}
Jul 24 00:41:04.210094 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:04.210072    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:04.224726 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:04.224671    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4" podStartSLOduration=1.2246544990000001 podStartE2EDuration="1.224654499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:41:04.223465014 +0000 UTC m=+1644.791281166" watchObservedRunningTime="2026-07-24 00:41:04.224654499 +0000 UTC m=+1644.792470649"
Jul 24 00:41:05.217162 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:05.217081    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv" event={"ID":"d8985c15-4674-4b6c-a42a-b9f77d2b57a0","Type":"ContainerStarted","Data":"7cb8411d692144083de15241161dd8d4743bbbdcd5bc8033941ae6b698b5bb0a"}
Jul 24 00:41:05.233732 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:05.233678    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv" podStartSLOduration=2.641198411 podStartE2EDuration="3.233660631s" totalImagesPullingTime="592.46222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:41:05.232423438 +0000 UTC m=+1645.800239590" watchObservedRunningTime="2026-07-24 00:41:05.233660631 +0000 UTC m=+1645.801476781"
Jul 24 00:41:05.234416 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:05.234386    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt" podStartSLOduration=2.234376138 podStartE2EDuration="2.234376138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:41:04.24074443 +0000 UTC m=+1644.808560581" watchObservedRunningTime="2026-07-24 00:41:05.234376138 +0000 UTC m=+1645.802192287"
Jul 24 00:41:06.221912 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:06.221874    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:12.231731 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:12.231700    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:15.219478 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:15.219443    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:15.219865 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:15.219531    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:38.263906 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.263870    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"]
Jul 24 00:41:38.264451 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.264134    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv" podUID="d8985c15-4674-4b6c-a42a-b9f77d2b57a0" containerName="maas-api" containerID="cri-o://7cb8411d692144083de15241161dd8d4743bbbdcd5bc8033941ae6b698b5bb0a" gracePeriod=30
Jul 24 00:41:38.324708 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.324672    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"]
Jul 24 00:41:38.325009 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.324964    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4" podUID="e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568" containerName="payload-processing" containerID="cri-o://b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549" gracePeriod=30
Jul 24 00:41:38.330183 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.330153    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"]
Jul 24 00:41:38.330406 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.330364    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt" podUID="c1844e56-5be6-46ea-9e56-1cf23cd9d6b8" containerName="payload-pre-processing" containerID="cri-o://c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579" gracePeriod=30
Jul 24 00:41:38.378156 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.378118    2560 generic.go:356] "Generic (PLEG): container finished" podID="d8985c15-4674-4b6c-a42a-b9f77d2b57a0" containerID="7cb8411d692144083de15241161dd8d4743bbbdcd5bc8033941ae6b698b5bb0a" exitCode=0
Jul 24 00:41:38.378288 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.378185    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv" event={"ID":"d8985c15-4674-4b6c-a42a-b9f77d2b57a0","Type":"ContainerDied","Data":"7cb8411d692144083de15241161dd8d4743bbbdcd5bc8033941ae6b698b5bb0a"}
Jul 24 00:41:38.600313 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.600282    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:38.603066 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.603046    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:38.624778 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.624751    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:38.711554 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.711521    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-maas-api-tls\") pod \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") "
Jul 24 00:41:38.711740 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.711606    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7ndv\" (UniqueName: \"kubernetes.io/projected/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-kube-api-access-h7ndv\") pod \"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568\" (UID: \"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568\") "
Jul 24 00:41:38.711740 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.711628    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-openshift-service-ca\") pod \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") "
Jul 24 00:41:38.711740 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.711661    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-plugins-config-volume\") pod \"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8\" (UID: \"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8\") "
Jul 24 00:41:38.711740 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.711690    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhlbr\" (UniqueName: \"kubernetes.io/projected/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-kube-api-access-dhlbr\") pod \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\" (UID: \"d8985c15-4674-4b6c-a42a-b9f77d2b57a0\") "
Jul 24 00:41:38.711740 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.711717    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-plugins-config-volume\") pod \"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568\" (UID: \"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568\") "
Jul 24 00:41:38.712061 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.711742    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mmg4\" (UniqueName: \"kubernetes.io/projected/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-kube-api-access-8mmg4\") pod \"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8\" (UID: \"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8\") "
Jul 24 00:41:38.712120 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.712066    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8985c15-4674-4b6c-a42a-b9f77d2b57a0" (UID: "d8985c15-4674-4b6c-a42a-b9f77d2b57a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:41:38.712120 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.712072    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1844e56-5be6-46ea-9e56-1cf23cd9d6b8" (UID: "c1844e56-5be6-46ea-9e56-1cf23cd9d6b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:41:38.712214 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.712177    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568" (UID: "e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:41:38.713766 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.713730    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8985c15-4674-4b6c-a42a-b9f77d2b57a0" (UID: "d8985c15-4674-4b6c-a42a-b9f77d2b57a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:41:38.713989 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.713822    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-kube-api-access-dhlbr" (OuterVolumeSpecName: "kube-api-access-dhlbr") pod "d8985c15-4674-4b6c-a42a-b9f77d2b57a0" (UID: "d8985c15-4674-4b6c-a42a-b9f77d2b57a0"). InnerVolumeSpecName "kube-api-access-dhlbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:41:38.714099 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.714074    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-kube-api-access-8mmg4" (OuterVolumeSpecName: "kube-api-access-8mmg4") pod "c1844e56-5be6-46ea-9e56-1cf23cd9d6b8" (UID: "c1844e56-5be6-46ea-9e56-1cf23cd9d6b8"). InnerVolumeSpecName "kube-api-access-8mmg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:41:38.714258 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.714236    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-kube-api-access-h7ndv" (OuterVolumeSpecName: "kube-api-access-h7ndv") pod "e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568" (UID: "e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568"). InnerVolumeSpecName "kube-api-access-h7ndv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:41:38.812728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.812647    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:41:38.812728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.812683    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7ndv\" (UniqueName: \"kubernetes.io/projected/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-kube-api-access-h7ndv\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:41:38.812728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.812699    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:41:38.812728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.812713    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:41:38.812728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.812727    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhlbr\" (UniqueName: \"kubernetes.io/projected/d8985c15-4674-4b6c-a42a-b9f77d2b57a0-kube-api-access-dhlbr\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:41:38.813017 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.812740    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:41:38.813017 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:38.812753    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mmg4\" (UniqueName: \"kubernetes.io/projected/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8-kube-api-access-8mmg4\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:41:39.384077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.384044    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"
Jul 24 00:41:39.384490 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.384053    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv" event={"ID":"d8985c15-4674-4b6c-a42a-b9f77d2b57a0","Type":"ContainerDied","Data":"fe805fbf169c48b7dee2f2f55a8d7558c17b4bf22e9cf7016fceddd23a1df586"}
Jul 24 00:41:39.384490 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.384181    2560 scope.go:117] "RemoveContainer" containerID="7cb8411d692144083de15241161dd8d4743bbbdcd5bc8033941ae6b698b5bb0a"
Jul 24 00:41:39.385573 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.385529    2560 generic.go:356] "Generic (PLEG): container finished" podID="e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568" containerID="b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549" exitCode=0
Jul 24 00:41:39.385650 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.385567    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"
Jul 24 00:41:39.385650 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.385589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4" event={"ID":"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568","Type":"ContainerDied","Data":"b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549"}
Jul 24 00:41:39.385748 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.385655    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4" event={"ID":"e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568","Type":"ContainerDied","Data":"ce219d7e3097673a99cef53c4057f2819877cf9e723dcea2f3e911af33b04e2a"}
Jul 24 00:41:39.387150 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.387124    2560 generic.go:356] "Generic (PLEG): container finished" podID="c1844e56-5be6-46ea-9e56-1cf23cd9d6b8" containerID="c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579" exitCode=0
Jul 24 00:41:39.387275 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.387176    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"
Jul 24 00:41:39.387275 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.387190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt" event={"ID":"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8","Type":"ContainerDied","Data":"c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579"}
Jul 24 00:41:39.387275 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.387244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt" event={"ID":"c1844e56-5be6-46ea-9e56-1cf23cd9d6b8","Type":"ContainerDied","Data":"2d07424e8f3bcbb7806c11f4d36205d06a5a4844eb3582c094d28994d59be107"}
Jul 24 00:41:39.394070 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.394054    2560 scope.go:117] "RemoveContainer" containerID="b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549"
Jul 24 00:41:39.402373 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.402356    2560 scope.go:117] "RemoveContainer" containerID="b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549"
Jul 24 00:41:39.402637 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:41:39.402618    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549\": container with ID starting with b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549 not found: ID does not exist" containerID="b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549"
Jul 24 00:41:39.402678 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.402645    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549"} err="failed to get container status \"b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549\": rpc error: code = NotFound desc = could not find container \"b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549\": container with ID starting with b2ff565c2435b96eb4d363b60c62c044da0e9dcb09e162155c38a6f528a24549 not found: ID does not exist"
Jul 24 00:41:39.402678 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.402661    2560 scope.go:117] "RemoveContainer" containerID="c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579"
Jul 24 00:41:39.410541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.410519    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"]
Jul 24 00:41:39.410760 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.410741    2560 scope.go:117] "RemoveContainer" containerID="c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579"
Jul 24 00:41:39.411054 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:41:39.411032    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579\": container with ID starting with c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579 not found: ID does not exist" containerID="c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579"
Jul 24 00:41:39.411153 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.411066    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579"} err="failed to get container status \"c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579\": rpc error: code = NotFound desc = could not find container \"c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579\": container with ID starting with c9e33dd442f27653f708efc6ca56546d4ddd6cb0c4e48176a79bc8bff1acd579 not found: ID does not exist"
Jul 24 00:41:39.413945 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.413925    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eaba160e-7889b5d4b6-78hvv"]
Jul 24 00:41:39.423281 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.423258    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"]
Jul 24 00:41:39.424818 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.424799    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-eaba160e-54469f6f4f-lbhp4"]
Jul 24 00:41:39.433672 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.433650    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"]
Jul 24 00:41:39.437300 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.437278    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-eaba160e-57b77cfb76-qg2nt"]
Jul 24 00:41:39.967290 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.967252    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1844e56-5be6-46ea-9e56-1cf23cd9d6b8" path="/var/lib/kubelet/pods/c1844e56-5be6-46ea-9e56-1cf23cd9d6b8/volumes"
Jul 24 00:41:39.967643 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.967629    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8985c15-4674-4b6c-a42a-b9f77d2b57a0" path="/var/lib/kubelet/pods/d8985c15-4674-4b6c-a42a-b9f77d2b57a0/volumes"
Jul 24 00:41:39.967951 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:39.967939    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568" path="/var/lib/kubelet/pods/e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568/volumes"
Jul 24 00:41:46.047170 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.047138    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"]
Jul 24 00:41:46.047541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.047464    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8985c15-4674-4b6c-a42a-b9f77d2b57a0" containerName="maas-api"
Jul 24 00:41:46.047541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.047475    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8985c15-4674-4b6c-a42a-b9f77d2b57a0" containerName="maas-api"
Jul 24 00:41:46.047541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.047483    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1844e56-5be6-46ea-9e56-1cf23cd9d6b8" containerName="payload-pre-processing"
Jul 24 00:41:46.047541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.047489    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1844e56-5be6-46ea-9e56-1cf23cd9d6b8" containerName="payload-pre-processing"
Jul 24 00:41:46.047541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.047495    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568" containerName="payload-processing"
Jul 24 00:41:46.047541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.047501    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568" containerName="payload-processing"
Jul 24 00:41:46.047818 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.047571    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1844e56-5be6-46ea-9e56-1cf23cd9d6b8" containerName="payload-pre-processing"
Jul 24 00:41:46.047818 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.047580    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ec5c73-8f7e-485c-b6e6-ab6f9bdcf568" containerName="payload-processing"
Jul 24 00:41:46.047818 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.047588    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8985c15-4674-4b6c-a42a-b9f77d2b57a0" containerName="maas-api"
Jul 24 00:41:46.050663 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.050639    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.052999 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.052971    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a227ad09-serving-cert\""
Jul 24 00:41:46.062304 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.062277    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"]
Jul 24 00:41:46.172253 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.172220    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d7pd\" (UniqueName: \"kubernetes.io/projected/79b0ba00-242b-490a-a7a9-c82e25e66a75-kube-api-access-8d7pd\") pod \"maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.172427 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.172271    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b0ba00-242b-490a-a7a9-c82e25e66a75-maas-api-tls\") pod \"maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.172427 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.172405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b0ba00-242b-490a-a7a9-c82e25e66a75-openshift-service-ca\") pod \"maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.273368 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.273330    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b0ba00-242b-490a-a7a9-c82e25e66a75-openshift-service-ca\") pod \"maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.273368 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.273372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d7pd\" (UniqueName: \"kubernetes.io/projected/79b0ba00-242b-490a-a7a9-c82e25e66a75-kube-api-access-8d7pd\") pod \"maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.273603 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.273421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b0ba00-242b-490a-a7a9-c82e25e66a75-maas-api-tls\") pod \"maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.273661 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:41:46.273599    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-serving-cert: secret "maas-api-e2e-ait-a227ad09-serving-cert" not found
Jul 24 00:41:46.273708 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:41:46.273672    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b0ba00-242b-490a-a7a9-c82e25e66a75-maas-api-tls podName:79b0ba00-242b-490a-a7a9-c82e25e66a75 nodeName:}" failed. No retries permitted until 2026-07-24 00:41:46.773651191 +0000 UTC m=+1687.341467319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79b0ba00-242b-490a-a7a9-c82e25e66a75-maas-api-tls") pod "maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v" (UID: "79b0ba00-242b-490a-a7a9-c82e25e66a75") : secret "maas-api-e2e-ait-a227ad09-serving-cert" not found
Jul 24 00:41:46.274084 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.274063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b0ba00-242b-490a-a7a9-c82e25e66a75-openshift-service-ca\") pod \"maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.282166 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.282136    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d7pd\" (UniqueName: \"kubernetes.io/projected/79b0ba00-242b-490a-a7a9-c82e25e66a75-kube-api-access-8d7pd\") pod \"maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.723223 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.723184    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"]
Jul 24 00:41:46.726727 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.726702    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:41:46.729383 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.729361    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a227ad09-dockercfg-xwgnm\""
Jul 24 00:41:46.729482 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.729359    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a227ad09\""
Jul 24 00:41:46.737867 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.737822    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"]
Jul 24 00:41:46.745345 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.745318    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"]
Jul 24 00:41:46.749335 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.749314    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:41:46.759162 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.759137    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"]
Jul 24 00:41:46.778555 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.778522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98656008-827c-4857-a902-ddb487d7aef2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j\" (UID: \"98656008-827c-4857-a902-ddb487d7aef2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:41:46.778677 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.778635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfnmq\" (UniqueName: \"kubernetes.io/projected/98656008-827c-4857-a902-ddb487d7aef2-kube-api-access-mfnmq\") pod \"payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j\" (UID: \"98656008-827c-4857-a902-ddb487d7aef2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:41:46.778777 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.778760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b0ba00-242b-490a-a7a9-c82e25e66a75-maas-api-tls\") pod \"maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.781179 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.781159    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b0ba00-242b-490a-a7a9-c82e25e66a75-maas-api-tls\") pod \"maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.879255 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.879216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfnmq\" (UniqueName: \"kubernetes.io/projected/98656008-827c-4857-a902-ddb487d7aef2-kube-api-access-mfnmq\") pod \"payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j\" (UID: \"98656008-827c-4857-a902-ddb487d7aef2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:41:46.879403 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.879265    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv9lr\" (UniqueName: \"kubernetes.io/projected/ff2c3a94-63d2-44df-b1ba-93df08073922-kube-api-access-hv9lr\") pod \"payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h\" (UID: \"ff2c3a94-63d2-44df-b1ba-93df08073922\") " pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:41:46.879474 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.879401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98656008-827c-4857-a902-ddb487d7aef2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j\" (UID: \"98656008-827c-4857-a902-ddb487d7aef2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:41:46.879474 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.879446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2c3a94-63d2-44df-b1ba-93df08073922-plugins-config-volume\") pod \"payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h\" (UID: \"ff2c3a94-63d2-44df-b1ba-93df08073922\") " pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:41:46.880329 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.880310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98656008-827c-4857-a902-ddb487d7aef2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j\" (UID: \"98656008-827c-4857-a902-ddb487d7aef2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:41:46.888173 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.888149    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfnmq\" (UniqueName: \"kubernetes.io/projected/98656008-827c-4857-a902-ddb487d7aef2-kube-api-access-mfnmq\") pod \"payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j\" (UID: \"98656008-827c-4857-a902-ddb487d7aef2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:41:46.962032 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.961913    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:46.980382 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.980312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv9lr\" (UniqueName: \"kubernetes.io/projected/ff2c3a94-63d2-44df-b1ba-93df08073922-kube-api-access-hv9lr\") pod \"payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h\" (UID: \"ff2c3a94-63d2-44df-b1ba-93df08073922\") " pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:41:46.980512 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.980424    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2c3a94-63d2-44df-b1ba-93df08073922-plugins-config-volume\") pod \"payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h\" (UID: \"ff2c3a94-63d2-44df-b1ba-93df08073922\") " pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:41:46.981353 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.981325    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2c3a94-63d2-44df-b1ba-93df08073922-plugins-config-volume\") pod \"payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h\" (UID: \"ff2c3a94-63d2-44df-b1ba-93df08073922\") " pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:41:46.988279 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:46.988253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv9lr\" (UniqueName: \"kubernetes.io/projected/ff2c3a94-63d2-44df-b1ba-93df08073922-kube-api-access-hv9lr\") pod \"payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h\" (UID: \"ff2c3a94-63d2-44df-b1ba-93df08073922\") " pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:41:47.038355 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:47.038304    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:41:47.063460 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:47.063424    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:41:47.117429 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:47.116524    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"]
Jul 24 00:41:47.120545 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:41:47.120510    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b0ba00_242b_490a_a7a9_c82e25e66a75.slice/crio-d9fad6e3477cbb11acb8e5322e0e6b29f31e1803f072e87247023f07626bad56 WatchSource:0}: Error finding container d9fad6e3477cbb11acb8e5322e0e6b29f31e1803f072e87247023f07626bad56: Status 404 returned error can't find the container with id d9fad6e3477cbb11acb8e5322e0e6b29f31e1803f072e87247023f07626bad56
Jul 24 00:41:47.226227 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:47.226197    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"]
Jul 24 00:41:47.228271 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:41:47.228248    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2c3a94_63d2_44df_b1ba_93df08073922.slice/crio-2164e03dff4f1616f158bc16aa111b7255116d7b64c2e4c2062a9274077b7320 WatchSource:0}: Error finding container 2164e03dff4f1616f158bc16aa111b7255116d7b64c2e4c2062a9274077b7320: Status 404 returned error can't find the container with id 2164e03dff4f1616f158bc16aa111b7255116d7b64c2e4c2062a9274077b7320
Jul 24 00:41:47.405888 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:47.405859    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"]
Jul 24 00:41:47.408166 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:41:47.408138    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98656008_827c_4857_a902_ddb487d7aef2.slice/crio-6d5bf00f1e07989ea920bdecebd9b2a67c3768857b5c71a4188911b67d1fade0 WatchSource:0}: Error finding container 6d5bf00f1e07989ea920bdecebd9b2a67c3768857b5c71a4188911b67d1fade0: Status 404 returned error can't find the container with id 6d5bf00f1e07989ea920bdecebd9b2a67c3768857b5c71a4188911b67d1fade0
Jul 24 00:41:47.421940 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:47.421913    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v" event={"ID":"79b0ba00-242b-490a-a7a9-c82e25e66a75","Type":"ContainerStarted","Data":"d9fad6e3477cbb11acb8e5322e0e6b29f31e1803f072e87247023f07626bad56"}
Jul 24 00:41:47.426848 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:47.426800    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h" event={"ID":"ff2c3a94-63d2-44df-b1ba-93df08073922","Type":"ContainerStarted","Data":"50657f0ccf57f82a0b6ad841cd49faab254541a4a600789d0a2ccbd1df93c024"}
Jul 24 00:41:47.426949 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:47.426889    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h" event={"ID":"ff2c3a94-63d2-44df-b1ba-93df08073922","Type":"ContainerStarted","Data":"2164e03dff4f1616f158bc16aa111b7255116d7b64c2e4c2062a9274077b7320"}
Jul 24 00:41:47.427014 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:47.426954    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:41:47.428210 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:47.428190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j" event={"ID":"98656008-827c-4857-a902-ddb487d7aef2","Type":"ContainerStarted","Data":"6d5bf00f1e07989ea920bdecebd9b2a67c3768857b5c71a4188911b67d1fade0"}
Jul 24 00:41:47.444125 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:47.444074    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h" podStartSLOduration=1.444059767 podStartE2EDuration="1.444059767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:41:47.441332299 +0000 UTC m=+1688.009148440" watchObservedRunningTime="2026-07-24 00:41:47.444059767 +0000 UTC m=+1688.011875909"
Jul 24 00:41:48.436385 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:48.436348    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v" event={"ID":"79b0ba00-242b-490a-a7a9-c82e25e66a75","Type":"ContainerStarted","Data":"ea58bb74db25529df3f58543c4ea0940f644f2bb725387730965c40b14da6926"}
Jul 24 00:41:48.436866 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:48.436422    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:48.438407 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:48.438345    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j" event={"ID":"98656008-827c-4857-a902-ddb487d7aef2","Type":"ContainerStarted","Data":"95fc1aad1754047cf19ffb56a06176a82319131bb12014c1c0182289c9852a1b"}
Jul 24 00:41:48.438544 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:48.438466    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:41:48.460602 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:48.460536    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v" podStartSLOduration=2.046241729 podStartE2EDuration="2.460517477s" totalImagesPullingTime="414.275748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:41:48.460395019 +0000 UTC m=+1689.028211170" watchObservedRunningTime="2026-07-24 00:41:48.460517477 +0000 UTC m=+1689.028333631"
Jul 24 00:41:48.494616 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:48.494562    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j" podStartSLOduration=2.494546434 podStartE2EDuration="2.494546434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:41:48.493115082 +0000 UTC m=+1689.060931256" watchObservedRunningTime="2026-07-24 00:41:48.494546434 +0000 UTC m=+1689.062362585"
Jul 24 00:41:54.450283 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:54.450249    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:41:58.440690 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:58.440656    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:41:59.446359 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:41:59.446319    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:42:22.374145 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.374060    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"]
Jul 24 00:42:22.374616 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.374344    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v" podUID="79b0ba00-242b-490a-a7a9-c82e25e66a75" containerName="maas-api" containerID="cri-o://ea58bb74db25529df3f58543c4ea0940f644f2bb725387730965c40b14da6926" gracePeriod=30
Jul 24 00:42:22.424547 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.424511    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"]
Jul 24 00:42:22.424858 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.424786    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h" podUID="ff2c3a94-63d2-44df-b1ba-93df08073922" containerName="payload-processing" containerID="cri-o://50657f0ccf57f82a0b6ad841cd49faab254541a4a600789d0a2ccbd1df93c024" gracePeriod=30
Jul 24 00:42:22.431688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.431659    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"]
Jul 24 00:42:22.432079 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.432030    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j" podUID="98656008-827c-4857-a902-ddb487d7aef2" containerName="payload-pre-processing" containerID="cri-o://95fc1aad1754047cf19ffb56a06176a82319131bb12014c1c0182289c9852a1b" gracePeriod=30
Jul 24 00:42:22.606749 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.606719    2560 generic.go:356] "Generic (PLEG): container finished" podID="79b0ba00-242b-490a-a7a9-c82e25e66a75" containerID="ea58bb74db25529df3f58543c4ea0940f644f2bb725387730965c40b14da6926" exitCode=0
Jul 24 00:42:22.606906 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.606800    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v" event={"ID":"79b0ba00-242b-490a-a7a9-c82e25e66a75","Type":"ContainerDied","Data":"ea58bb74db25529df3f58543c4ea0940f644f2bb725387730965c40b14da6926"}
Jul 24 00:42:22.608633 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.608278    2560 generic.go:356] "Generic (PLEG): container finished" podID="ff2c3a94-63d2-44df-b1ba-93df08073922" containerID="50657f0ccf57f82a0b6ad841cd49faab254541a4a600789d0a2ccbd1df93c024" exitCode=0
Jul 24 00:42:22.608633 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.608341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h" event={"ID":"ff2c3a94-63d2-44df-b1ba-93df08073922","Type":"ContainerDied","Data":"50657f0ccf57f82a0b6ad841cd49faab254541a4a600789d0a2ccbd1df93c024"}
Jul 24 00:42:22.609795 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.609773    2560 generic.go:356] "Generic (PLEG): container finished" podID="98656008-827c-4857-a902-ddb487d7aef2" containerID="95fc1aad1754047cf19ffb56a06176a82319131bb12014c1c0182289c9852a1b" exitCode=0
Jul 24 00:42:22.609925 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.609852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j" event={"ID":"98656008-827c-4857-a902-ddb487d7aef2","Type":"ContainerDied","Data":"95fc1aad1754047cf19ffb56a06176a82319131bb12014c1c0182289c9852a1b"}
Jul 24 00:42:22.715451 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.715426    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:42:22.723377 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.723354    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:42:22.734154 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.734128    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:42:22.800788 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.800745    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d7pd\" (UniqueName: \"kubernetes.io/projected/79b0ba00-242b-490a-a7a9-c82e25e66a75-kube-api-access-8d7pd\") pod \"79b0ba00-242b-490a-a7a9-c82e25e66a75\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") "
Jul 24 00:42:22.800788 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.800788    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfnmq\" (UniqueName: \"kubernetes.io/projected/98656008-827c-4857-a902-ddb487d7aef2-kube-api-access-mfnmq\") pod \"98656008-827c-4857-a902-ddb487d7aef2\" (UID: \"98656008-827c-4857-a902-ddb487d7aef2\") "
Jul 24 00:42:22.801049 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.800848    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b0ba00-242b-490a-a7a9-c82e25e66a75-maas-api-tls\") pod \"79b0ba00-242b-490a-a7a9-c82e25e66a75\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") "
Jul 24 00:42:22.801049 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.800867    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b0ba00-242b-490a-a7a9-c82e25e66a75-openshift-service-ca\") pod \"79b0ba00-242b-490a-a7a9-c82e25e66a75\" (UID: \"79b0ba00-242b-490a-a7a9-c82e25e66a75\") "
Jul 24 00:42:22.801049 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.800911    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2c3a94-63d2-44df-b1ba-93df08073922-plugins-config-volume\") pod \"ff2c3a94-63d2-44df-b1ba-93df08073922\" (UID: \"ff2c3a94-63d2-44df-b1ba-93df08073922\") "
Jul 24 00:42:22.801049 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.800948    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98656008-827c-4857-a902-ddb487d7aef2-plugins-config-volume\") pod \"98656008-827c-4857-a902-ddb487d7aef2\" (UID: \"98656008-827c-4857-a902-ddb487d7aef2\") "
Jul 24 00:42:22.801049 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.800974    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv9lr\" (UniqueName: \"kubernetes.io/projected/ff2c3a94-63d2-44df-b1ba-93df08073922-kube-api-access-hv9lr\") pod \"ff2c3a94-63d2-44df-b1ba-93df08073922\" (UID: \"ff2c3a94-63d2-44df-b1ba-93df08073922\") "
Jul 24 00:42:22.801339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.801308    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b0ba00-242b-490a-a7a9-c82e25e66a75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79b0ba00-242b-490a-a7a9-c82e25e66a75" (UID: "79b0ba00-242b-490a-a7a9-c82e25e66a75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:42:22.801399 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.801331    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2c3a94-63d2-44df-b1ba-93df08073922-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff2c3a94-63d2-44df-b1ba-93df08073922" (UID: "ff2c3a94-63d2-44df-b1ba-93df08073922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:42:22.801399 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.801390    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98656008-827c-4857-a902-ddb487d7aef2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98656008-827c-4857-a902-ddb487d7aef2" (UID: "98656008-827c-4857-a902-ddb487d7aef2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:42:22.803050 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.803020    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b0ba00-242b-490a-a7a9-c82e25e66a75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79b0ba00-242b-490a-a7a9-c82e25e66a75" (UID: "79b0ba00-242b-490a-a7a9-c82e25e66a75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:42:22.803158 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.803054    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98656008-827c-4857-a902-ddb487d7aef2-kube-api-access-mfnmq" (OuterVolumeSpecName: "kube-api-access-mfnmq") pod "98656008-827c-4857-a902-ddb487d7aef2" (UID: "98656008-827c-4857-a902-ddb487d7aef2"). InnerVolumeSpecName "kube-api-access-mfnmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:42:22.803158 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.803094    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2c3a94-63d2-44df-b1ba-93df08073922-kube-api-access-hv9lr" (OuterVolumeSpecName: "kube-api-access-hv9lr") pod "ff2c3a94-63d2-44df-b1ba-93df08073922" (UID: "ff2c3a94-63d2-44df-b1ba-93df08073922"). InnerVolumeSpecName "kube-api-access-hv9lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:42:22.803158 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.803098    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b0ba00-242b-490a-a7a9-c82e25e66a75-kube-api-access-8d7pd" (OuterVolumeSpecName: "kube-api-access-8d7pd") pod "79b0ba00-242b-490a-a7a9-c82e25e66a75" (UID: "79b0ba00-242b-490a-a7a9-c82e25e66a75"). InnerVolumeSpecName "kube-api-access-8d7pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:42:22.902073 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.901993    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d7pd\" (UniqueName: \"kubernetes.io/projected/79b0ba00-242b-490a-a7a9-c82e25e66a75-kube-api-access-8d7pd\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:42:22.902073 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.902021    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfnmq\" (UniqueName: \"kubernetes.io/projected/98656008-827c-4857-a902-ddb487d7aef2-kube-api-access-mfnmq\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:42:22.902073 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.902030    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b0ba00-242b-490a-a7a9-c82e25e66a75-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:42:22.902073 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.902040    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b0ba00-242b-490a-a7a9-c82e25e66a75-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:42:22.902073 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.902048    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2c3a94-63d2-44df-b1ba-93df08073922-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:42:22.902073 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.902056    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98656008-827c-4857-a902-ddb487d7aef2-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:42:22.902073 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:22.902064    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv9lr\" (UniqueName: \"kubernetes.io/projected/ff2c3a94-63d2-44df-b1ba-93df08073922-kube-api-access-hv9lr\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:42:23.615648 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.615620    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"
Jul 24 00:42:23.616078 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.615620    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v" event={"ID":"79b0ba00-242b-490a-a7a9-c82e25e66a75","Type":"ContainerDied","Data":"d9fad6e3477cbb11acb8e5322e0e6b29f31e1803f072e87247023f07626bad56"}
Jul 24 00:42:23.616078 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.615753    2560 scope.go:117] "RemoveContainer" containerID="ea58bb74db25529df3f58543c4ea0940f644f2bb725387730965c40b14da6926"
Jul 24 00:42:23.617062 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.617018    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"
Jul 24 00:42:23.617127 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.617050    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h" event={"ID":"ff2c3a94-63d2-44df-b1ba-93df08073922","Type":"ContainerDied","Data":"2164e03dff4f1616f158bc16aa111b7255116d7b64c2e4c2062a9274077b7320"}
Jul 24 00:42:23.618523 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.618499    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j" event={"ID":"98656008-827c-4857-a902-ddb487d7aef2","Type":"ContainerDied","Data":"6d5bf00f1e07989ea920bdecebd9b2a67c3768857b5c71a4188911b67d1fade0"}
Jul 24 00:42:23.618632 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.618503    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"
Jul 24 00:42:23.625413 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.625396    2560 scope.go:117] "RemoveContainer" containerID="50657f0ccf57f82a0b6ad841cd49faab254541a4a600789d0a2ccbd1df93c024"
Jul 24 00:42:23.634551 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.634532    2560 scope.go:117] "RemoveContainer" containerID="95fc1aad1754047cf19ffb56a06176a82319131bb12014c1c0182289c9852a1b"
Jul 24 00:42:23.642469 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.642445    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"]
Jul 24 00:42:23.645664 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.645641    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a227ad09-7c68fb5ffb-6kj7v"]
Jul 24 00:42:23.656926 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.656900    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"]
Jul 24 00:42:23.660503 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.660480    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a227ad09-55f7659f8d-ztc5h"]
Jul 24 00:42:23.669351 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.669328    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"]
Jul 24 00:42:23.673158 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.673126    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a227ad09-fd4668948-zlf5j"]
Jul 24 00:42:23.969788 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.969748    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b0ba00-242b-490a-a7a9-c82e25e66a75" path="/var/lib/kubelet/pods/79b0ba00-242b-490a-a7a9-c82e25e66a75/volumes"
Jul 24 00:42:23.970181 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.970164    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98656008-827c-4857-a902-ddb487d7aef2" path="/var/lib/kubelet/pods/98656008-827c-4857-a902-ddb487d7aef2/volumes"
Jul 24 00:42:23.970478 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:23.970466    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff2c3a94-63d2-44df-b1ba-93df08073922" path="/var/lib/kubelet/pods/ff2c3a94-63d2-44df-b1ba-93df08073922/volumes"
Jul 24 00:42:31.412850 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.412799    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"]
Jul 24 00:42:31.413339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.413141    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98656008-827c-4857-a902-ddb487d7aef2" containerName="payload-pre-processing"
Jul 24 00:42:31.413339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.413153    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="98656008-827c-4857-a902-ddb487d7aef2" containerName="payload-pre-processing"
Jul 24 00:42:31.413339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.413171    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff2c3a94-63d2-44df-b1ba-93df08073922" containerName="payload-processing"
Jul 24 00:42:31.413339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.413177    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff2c3a94-63d2-44df-b1ba-93df08073922" containerName="payload-processing"
Jul 24 00:42:31.413339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.413189    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b0ba00-242b-490a-a7a9-c82e25e66a75" containerName="maas-api"
Jul 24 00:42:31.413339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.413195    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b0ba00-242b-490a-a7a9-c82e25e66a75" containerName="maas-api"
Jul 24 00:42:31.413339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.413250    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="98656008-827c-4857-a902-ddb487d7aef2" containerName="payload-pre-processing"
Jul 24 00:42:31.413339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.413259    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff2c3a94-63d2-44df-b1ba-93df08073922" containerName="payload-processing"
Jul 24 00:42:31.413339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.413266    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b0ba00-242b-490a-a7a9-c82e25e66a75" containerName="maas-api"
Jul 24 00:42:31.417786 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.417761    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:31.420428 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.420400    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-601239cb-serving-cert\""
Jul 24 00:42:31.427320 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.427096    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"]
Jul 24 00:42:31.478632 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.478589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6d7r\" (UniqueName: \"kubernetes.io/projected/b2d5e314-8c87-489a-8f05-211c5abb5310-kube-api-access-r6d7r\") pod \"maas-api-e2e-derive-601239cb-744f54c88c-pcq4w\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:31.478799 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.478698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d5e314-8c87-489a-8f05-211c5abb5310-openshift-service-ca\") pod \"maas-api-e2e-derive-601239cb-744f54c88c-pcq4w\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:31.478799 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.478734    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d5e314-8c87-489a-8f05-211c5abb5310-maas-api-tls\") pod \"maas-api-e2e-derive-601239cb-744f54c88c-pcq4w\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:31.579184 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.579149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d5e314-8c87-489a-8f05-211c5abb5310-openshift-service-ca\") pod \"maas-api-e2e-derive-601239cb-744f54c88c-pcq4w\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:31.579184 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.579188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d5e314-8c87-489a-8f05-211c5abb5310-maas-api-tls\") pod \"maas-api-e2e-derive-601239cb-744f54c88c-pcq4w\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:31.579375 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.579237    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6d7r\" (UniqueName: \"kubernetes.io/projected/b2d5e314-8c87-489a-8f05-211c5abb5310-kube-api-access-r6d7r\") pod \"maas-api-e2e-derive-601239cb-744f54c88c-pcq4w\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:31.579418 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:42:31.579386    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-serving-cert: secret "maas-api-e2e-derive-601239cb-serving-cert" not found
Jul 24 00:42:31.579491 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:42:31.579479    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2d5e314-8c87-489a-8f05-211c5abb5310-maas-api-tls podName:b2d5e314-8c87-489a-8f05-211c5abb5310 nodeName:}" failed. No retries permitted until 2026-07-24 00:42:32.079456215 +0000 UTC m=+1732.647272357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2d5e314-8c87-489a-8f05-211c5abb5310-maas-api-tls") pod "maas-api-e2e-derive-601239cb-744f54c88c-pcq4w" (UID: "b2d5e314-8c87-489a-8f05-211c5abb5310") : secret "maas-api-e2e-derive-601239cb-serving-cert" not found
Jul 24 00:42:31.579775 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.579756    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d5e314-8c87-489a-8f05-211c5abb5310-openshift-service-ca\") pod \"maas-api-e2e-derive-601239cb-744f54c88c-pcq4w\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:31.590595 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:31.590565    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6d7r\" (UniqueName: \"kubernetes.io/projected/b2d5e314-8c87-489a-8f05-211c5abb5310-kube-api-access-r6d7r\") pod \"maas-api-e2e-derive-601239cb-744f54c88c-pcq4w\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:32.083942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.083891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d5e314-8c87-489a-8f05-211c5abb5310-maas-api-tls\") pod \"maas-api-e2e-derive-601239cb-744f54c88c-pcq4w\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:32.086679 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.086645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d5e314-8c87-489a-8f05-211c5abb5310-maas-api-tls\") pod \"maas-api-e2e-derive-601239cb-744f54c88c-pcq4w\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:32.226275 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.223230    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"]
Jul 24 00:42:32.228657 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.228622    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:42:32.231804 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.231777    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-601239cb\""
Jul 24 00:42:32.237544 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.237405    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"]
Jul 24 00:42:32.270210 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.270180    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"]
Jul 24 00:42:32.273558 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.273540    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:42:32.275888 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.275867    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-601239cb-dockercfg-4gcw4\""
Jul 24 00:42:32.284123 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.284100    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"]
Jul 24 00:42:32.285219 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.285192    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bd6ab0-800c-4480-9da1-529a116bb01b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr\" (UID: \"c5bd6ab0-800c-4480-9da1-529a116bb01b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:42:32.285343 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.285312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vztjq\" (UniqueName: \"kubernetes.io/projected/c5bd6ab0-800c-4480-9da1-529a116bb01b-kube-api-access-vztjq\") pod \"payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr\" (UID: \"c5bd6ab0-800c-4480-9da1-529a116bb01b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:42:32.330676 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.330645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:32.386355 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.386315    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4sl8\" (UniqueName: \"kubernetes.io/projected/90fabf6e-ca71-4235-a900-8635d04b6db3-kube-api-access-j4sl8\") pod \"payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4\" (UID: \"90fabf6e-ca71-4235-a900-8635d04b6db3\") " pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:42:32.386494 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.386394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vztjq\" (UniqueName: \"kubernetes.io/projected/c5bd6ab0-800c-4480-9da1-529a116bb01b-kube-api-access-vztjq\") pod \"payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr\" (UID: \"c5bd6ab0-800c-4480-9da1-529a116bb01b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:42:32.386494 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.386479    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bd6ab0-800c-4480-9da1-529a116bb01b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr\" (UID: \"c5bd6ab0-800c-4480-9da1-529a116bb01b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:42:32.386633 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.386515    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fabf6e-ca71-4235-a900-8635d04b6db3-plugins-config-volume\") pod \"payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4\" (UID: \"90fabf6e-ca71-4235-a900-8635d04b6db3\") " pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:42:32.387465 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.387436    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bd6ab0-800c-4480-9da1-529a116bb01b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr\" (UID: \"c5bd6ab0-800c-4480-9da1-529a116bb01b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:42:32.397223 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.397169    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vztjq\" (UniqueName: \"kubernetes.io/projected/c5bd6ab0-800c-4480-9da1-529a116bb01b-kube-api-access-vztjq\") pod \"payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr\" (UID: \"c5bd6ab0-800c-4480-9da1-529a116bb01b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:42:32.470256 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.470205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"]
Jul 24 00:42:32.472452 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:42:32.472423    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d5e314_8c87_489a_8f05_211c5abb5310.slice/crio-3cb9f86463c0c99f3c4c0cd99f971421ea9829495dff006fa3b1cbece3d2a91d WatchSource:0}: Error finding container 3cb9f86463c0c99f3c4c0cd99f971421ea9829495dff006fa3b1cbece3d2a91d: Status 404 returned error can't find the container with id 3cb9f86463c0c99f3c4c0cd99f971421ea9829495dff006fa3b1cbece3d2a91d
Jul 24 00:42:32.487027 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.487003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4sl8\" (UniqueName: \"kubernetes.io/projected/90fabf6e-ca71-4235-a900-8635d04b6db3-kube-api-access-j4sl8\") pod \"payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4\" (UID: \"90fabf6e-ca71-4235-a900-8635d04b6db3\") " pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:42:32.487141 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.487081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fabf6e-ca71-4235-a900-8635d04b6db3-plugins-config-volume\") pod \"payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4\" (UID: \"90fabf6e-ca71-4235-a900-8635d04b6db3\") " pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:42:32.487716 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.487699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fabf6e-ca71-4235-a900-8635d04b6db3-plugins-config-volume\") pod \"payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4\" (UID: \"90fabf6e-ca71-4235-a900-8635d04b6db3\") " pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:42:32.494669 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.494648    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4sl8\" (UniqueName: \"kubernetes.io/projected/90fabf6e-ca71-4235-a900-8635d04b6db3-kube-api-access-j4sl8\") pod \"payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4\" (UID: \"90fabf6e-ca71-4235-a900-8635d04b6db3\") " pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:42:32.542624 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.542590    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:42:32.583922 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.583883    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:42:32.662509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.662344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w" event={"ID":"b2d5e314-8c87-489a-8f05-211c5abb5310","Type":"ContainerStarted","Data":"3cb9f86463c0c99f3c4c0cd99f971421ea9829495dff006fa3b1cbece3d2a91d"}
Jul 24 00:42:32.697655 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.697247    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"]
Jul 24 00:42:32.699152 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:42:32.699119    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5bd6ab0_800c_4480_9da1_529a116bb01b.slice/crio-2addaaebda03391c007b90d0f7ddef13339e4d9ee444be049dc8240959bead69 WatchSource:0}: Error finding container 2addaaebda03391c007b90d0f7ddef13339e4d9ee444be049dc8240959bead69: Status 404 returned error can't find the container with id 2addaaebda03391c007b90d0f7ddef13339e4d9ee444be049dc8240959bead69
Jul 24 00:42:32.743711 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:32.743690    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"]
Jul 24 00:42:32.746059 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:42:32.746027    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fabf6e_ca71_4235_a900_8635d04b6db3.slice/crio-72b0488affd4f991d211614851ed351a320d9f8645a9fe84b01c8e0f7816bf24 WatchSource:0}: Error finding container 72b0488affd4f991d211614851ed351a320d9f8645a9fe84b01c8e0f7816bf24: Status 404 returned error can't find the container with id 72b0488affd4f991d211614851ed351a320d9f8645a9fe84b01c8e0f7816bf24
Jul 24 00:42:33.669174 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:33.669137    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w" event={"ID":"b2d5e314-8c87-489a-8f05-211c5abb5310","Type":"ContainerStarted","Data":"62f1d23fd983bfda4c6ed6e2740417b169195c4c97d6600768d358f23aec1dc8"}
Jul 24 00:42:33.669604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:33.669208    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:33.670930 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:33.670904    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4" event={"ID":"90fabf6e-ca71-4235-a900-8635d04b6db3","Type":"ContainerStarted","Data":"873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1"}
Jul 24 00:42:33.671056 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:33.670953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4" event={"ID":"90fabf6e-ca71-4235-a900-8635d04b6db3","Type":"ContainerStarted","Data":"72b0488affd4f991d211614851ed351a320d9f8645a9fe84b01c8e0f7816bf24"}
Jul 24 00:42:33.671056 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:33.670977    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:42:33.672687 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:33.672658    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr" event={"ID":"c5bd6ab0-800c-4480-9da1-529a116bb01b","Type":"ContainerStarted","Data":"8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e"}
Jul 24 00:42:33.672770 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:33.672710    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr" event={"ID":"c5bd6ab0-800c-4480-9da1-529a116bb01b","Type":"ContainerStarted","Data":"2addaaebda03391c007b90d0f7ddef13339e4d9ee444be049dc8240959bead69"}
Jul 24 00:42:33.672770 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:33.672732    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:42:33.685702 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:33.685662    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w" podStartSLOduration=2.093168582 podStartE2EDuration="2.685649068s" totalImagesPullingTime="592.480486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:42:33.68478199 +0000 UTC m=+1734.252598142" watchObservedRunningTime="2026-07-24 00:42:33.685649068 +0000 UTC m=+1734.253465219"
Jul 24 00:42:33.698234 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:33.698188    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr" podStartSLOduration=1.6981704199999998 podStartE2EDuration="1.69817042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:42:33.697939007 +0000 UTC m=+1734.265755204" watchObservedRunningTime="2026-07-24 00:42:33.69817042 +0000 UTC m=+1734.265986570"
Jul 24 00:42:33.713715 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:33.713668    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4" podStartSLOduration=1.713654234 podStartE2EDuration="1.713654234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:42:33.711893352 +0000 UTC m=+1734.279709502" watchObservedRunningTime="2026-07-24 00:42:33.713654234 +0000 UTC m=+1734.281470385"
Jul 24 00:42:39.683441 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:39.683409    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:42:44.679661 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:44.679629    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:42:44.680090 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:42:44.679704    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:43:06.707863 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:06.706906    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"]
Jul 24 00:43:06.707863 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:06.707152    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w" podUID="b2d5e314-8c87-489a-8f05-211c5abb5310" containerName="maas-api" containerID="cri-o://62f1d23fd983bfda4c6ed6e2740417b169195c4c97d6600768d358f23aec1dc8" gracePeriod=30
Jul 24 00:43:06.754622 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:06.754586    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"]
Jul 24 00:43:06.754861 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:06.754807    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4" podUID="90fabf6e-ca71-4235-a900-8635d04b6db3" containerName="payload-processing" containerID="cri-o://873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1" gracePeriod=30
Jul 24 00:43:06.763638 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:06.763607    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"]
Jul 24 00:43:06.763824 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:06.763804    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr" podUID="c5bd6ab0-800c-4480-9da1-529a116bb01b" containerName="payload-pre-processing" containerID="cri-o://8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e" gracePeriod=30
Jul 24 00:43:06.840119 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:06.840087    2560 generic.go:356] "Generic (PLEG): container finished" podID="b2d5e314-8c87-489a-8f05-211c5abb5310" containerID="62f1d23fd983bfda4c6ed6e2740417b169195c4c97d6600768d358f23aec1dc8" exitCode=0
Jul 24 00:43:06.840267 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:06.840148    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w" event={"ID":"b2d5e314-8c87-489a-8f05-211c5abb5310","Type":"ContainerDied","Data":"62f1d23fd983bfda4c6ed6e2740417b169195c4c97d6600768d358f23aec1dc8"}
Jul 24 00:43:07.013130 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.013100    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:43:07.035235 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.035206    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:43:07.046619 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.046593    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:43:07.188516 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.188478    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bd6ab0-800c-4480-9da1-529a116bb01b-plugins-config-volume\") pod \"c5bd6ab0-800c-4480-9da1-529a116bb01b\" (UID: \"c5bd6ab0-800c-4480-9da1-529a116bb01b\") "
Jul 24 00:43:07.188516 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.188522    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fabf6e-ca71-4235-a900-8635d04b6db3-plugins-config-volume\") pod \"90fabf6e-ca71-4235-a900-8635d04b6db3\" (UID: \"90fabf6e-ca71-4235-a900-8635d04b6db3\") "
Jul 24 00:43:07.188746 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.188552    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d5e314-8c87-489a-8f05-211c5abb5310-maas-api-tls\") pod \"b2d5e314-8c87-489a-8f05-211c5abb5310\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") "
Jul 24 00:43:07.188746 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.188703    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4sl8\" (UniqueName: \"kubernetes.io/projected/90fabf6e-ca71-4235-a900-8635d04b6db3-kube-api-access-j4sl8\") pod \"90fabf6e-ca71-4235-a900-8635d04b6db3\" (UID: \"90fabf6e-ca71-4235-a900-8635d04b6db3\") "
Jul 24 00:43:07.188887 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.188813    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d5e314-8c87-489a-8f05-211c5abb5310-openshift-service-ca\") pod \"b2d5e314-8c87-489a-8f05-211c5abb5310\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") "
Jul 24 00:43:07.188887 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.188871    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vztjq\" (UniqueName: \"kubernetes.io/projected/c5bd6ab0-800c-4480-9da1-529a116bb01b-kube-api-access-vztjq\") pod \"c5bd6ab0-800c-4480-9da1-529a116bb01b\" (UID: \"c5bd6ab0-800c-4480-9da1-529a116bb01b\") "
Jul 24 00:43:07.188988 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.188900    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6d7r\" (UniqueName: \"kubernetes.io/projected/b2d5e314-8c87-489a-8f05-211c5abb5310-kube-api-access-r6d7r\") pod \"b2d5e314-8c87-489a-8f05-211c5abb5310\" (UID: \"b2d5e314-8c87-489a-8f05-211c5abb5310\") "
Jul 24 00:43:07.189036 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.188992    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5bd6ab0-800c-4480-9da1-529a116bb01b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5bd6ab0-800c-4480-9da1-529a116bb01b" (UID: "c5bd6ab0-800c-4480-9da1-529a116bb01b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:43:07.189160 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.189109    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90fabf6e-ca71-4235-a900-8635d04b6db3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90fabf6e-ca71-4235-a900-8635d04b6db3" (UID: "90fabf6e-ca71-4235-a900-8635d04b6db3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:43:07.189273 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.189172    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d5e314-8c87-489a-8f05-211c5abb5310-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2d5e314-8c87-489a-8f05-211c5abb5310" (UID: "b2d5e314-8c87-489a-8f05-211c5abb5310"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:43:07.189273 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.189196    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bd6ab0-800c-4480-9da1-529a116bb01b-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:43:07.190936 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.190908    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d5e314-8c87-489a-8f05-211c5abb5310-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2d5e314-8c87-489a-8f05-211c5abb5310" (UID: "b2d5e314-8c87-489a-8f05-211c5abb5310"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:43:07.190936 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.190922    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fabf6e-ca71-4235-a900-8635d04b6db3-kube-api-access-j4sl8" (OuterVolumeSpecName: "kube-api-access-j4sl8") pod "90fabf6e-ca71-4235-a900-8635d04b6db3" (UID: "90fabf6e-ca71-4235-a900-8635d04b6db3"). InnerVolumeSpecName "kube-api-access-j4sl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:43:07.191067 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.190930    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d5e314-8c87-489a-8f05-211c5abb5310-kube-api-access-r6d7r" (OuterVolumeSpecName: "kube-api-access-r6d7r") pod "b2d5e314-8c87-489a-8f05-211c5abb5310" (UID: "b2d5e314-8c87-489a-8f05-211c5abb5310"). InnerVolumeSpecName "kube-api-access-r6d7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:43:07.191067 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.191009    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5bd6ab0-800c-4480-9da1-529a116bb01b-kube-api-access-vztjq" (OuterVolumeSpecName: "kube-api-access-vztjq") pod "c5bd6ab0-800c-4480-9da1-529a116bb01b" (UID: "c5bd6ab0-800c-4480-9da1-529a116bb01b"). InnerVolumeSpecName "kube-api-access-vztjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:43:07.290647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.290558    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d5e314-8c87-489a-8f05-211c5abb5310-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:43:07.290647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.290590    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vztjq\" (UniqueName: \"kubernetes.io/projected/c5bd6ab0-800c-4480-9da1-529a116bb01b-kube-api-access-vztjq\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:43:07.290647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.290599    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6d7r\" (UniqueName: \"kubernetes.io/projected/b2d5e314-8c87-489a-8f05-211c5abb5310-kube-api-access-r6d7r\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:43:07.290647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.290608    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fabf6e-ca71-4235-a900-8635d04b6db3-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:43:07.290647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.290618    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d5e314-8c87-489a-8f05-211c5abb5310-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:43:07.290647 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.290627    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4sl8\" (UniqueName: \"kubernetes.io/projected/90fabf6e-ca71-4235-a900-8635d04b6db3-kube-api-access-j4sl8\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:43:07.846622 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.846589    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"
Jul 24 00:43:07.847091 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.846594    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w" event={"ID":"b2d5e314-8c87-489a-8f05-211c5abb5310","Type":"ContainerDied","Data":"3cb9f86463c0c99f3c4c0cd99f971421ea9829495dff006fa3b1cbece3d2a91d"}
Jul 24 00:43:07.847091 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.846779    2560 scope.go:117] "RemoveContainer" containerID="62f1d23fd983bfda4c6ed6e2740417b169195c4c97d6600768d358f23aec1dc8"
Jul 24 00:43:07.848249 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.847892    2560 generic.go:356] "Generic (PLEG): container finished" podID="90fabf6e-ca71-4235-a900-8635d04b6db3" containerID="873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1" exitCode=0
Jul 24 00:43:07.848249 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.847939    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"
Jul 24 00:43:07.848249 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.847964    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4" event={"ID":"90fabf6e-ca71-4235-a900-8635d04b6db3","Type":"ContainerDied","Data":"873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1"}
Jul 24 00:43:07.848249 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.848021    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4" event={"ID":"90fabf6e-ca71-4235-a900-8635d04b6db3","Type":"ContainerDied","Data":"72b0488affd4f991d211614851ed351a320d9f8645a9fe84b01c8e0f7816bf24"}
Jul 24 00:43:07.849418 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.849394    2560 generic.go:356] "Generic (PLEG): container finished" podID="c5bd6ab0-800c-4480-9da1-529a116bb01b" containerID="8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e" exitCode=0
Jul 24 00:43:07.849536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.849423    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr" event={"ID":"c5bd6ab0-800c-4480-9da1-529a116bb01b","Type":"ContainerDied","Data":"8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e"}
Jul 24 00:43:07.849536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.849451    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"
Jul 24 00:43:07.849536 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.849466    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr" event={"ID":"c5bd6ab0-800c-4480-9da1-529a116bb01b","Type":"ContainerDied","Data":"2addaaebda03391c007b90d0f7ddef13339e4d9ee444be049dc8240959bead69"}
Jul 24 00:43:07.857049 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.857028    2560 scope.go:117] "RemoveContainer" containerID="873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1"
Jul 24 00:43:07.865540 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.865520    2560 scope.go:117] "RemoveContainer" containerID="873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1"
Jul 24 00:43:07.865760 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:43:07.865743    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1\": container with ID starting with 873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1 not found: ID does not exist" containerID="873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1"
Jul 24 00:43:07.865807 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.865767    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1"} err="failed to get container status \"873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1\": rpc error: code = NotFound desc = could not find container \"873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1\": container with ID starting with 873a1bb9a5c0b8413a5b5baa3333513539d5dc663662a9a70fe28e828abbe4b1 not found: ID does not exist"
Jul 24 00:43:07.865807 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.865783    2560 scope.go:117] "RemoveContainer" containerID="8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e"
Jul 24 00:43:07.873401 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.873340    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"]
Jul 24 00:43:07.873462 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.873419    2560 scope.go:117] "RemoveContainer" containerID="8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e"
Jul 24 00:43:07.873689 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:43:07.873670    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e\": container with ID starting with 8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e not found: ID does not exist" containerID="8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e"
Jul 24 00:43:07.873754 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.873698    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e"} err="failed to get container status \"8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e\": rpc error: code = NotFound desc = could not find container \"8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e\": container with ID starting with 8a0e07dc22bea0bb46f22cea96f38c575c7be4ec35ddecfde88e09f46e229d7e not found: ID does not exist"
Jul 24 00:43:07.877001 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.876976    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-601239cb-744f54c88c-pcq4w"]
Jul 24 00:43:07.886869 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.886845    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"]
Jul 24 00:43:07.890601 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.890579    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-601239cb-65cd8b4fb-x6gvr"]
Jul 24 00:43:07.900465 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.900440    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"]
Jul 24 00:43:07.902048 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.902027    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-601239cb-dd54c66f4-nvpn4"]
Jul 24 00:43:07.967475 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.967441    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90fabf6e-ca71-4235-a900-8635d04b6db3" path="/var/lib/kubelet/pods/90fabf6e-ca71-4235-a900-8635d04b6db3/volumes"
Jul 24 00:43:07.967849 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.967821    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2d5e314-8c87-489a-8f05-211c5abb5310" path="/var/lib/kubelet/pods/b2d5e314-8c87-489a-8f05-211c5abb5310/volumes"
Jul 24 00:43:07.968155 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:43:07.968143    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5bd6ab0-800c-4480-9da1-529a116bb01b" path="/var/lib/kubelet/pods/c5bd6ab0-800c-4480-9da1-529a116bb01b/volumes"
Jul 24 00:44:51.884901 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.884849    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"]
Jul 24 00:44:51.885397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.885242    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d5e314-8c87-489a-8f05-211c5abb5310" containerName="maas-api"
Jul 24 00:44:51.885397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.885256    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d5e314-8c87-489a-8f05-211c5abb5310" containerName="maas-api"
Jul 24 00:44:51.885397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.885267    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5bd6ab0-800c-4480-9da1-529a116bb01b" containerName="payload-pre-processing"
Jul 24 00:44:51.885397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.885272    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5bd6ab0-800c-4480-9da1-529a116bb01b" containerName="payload-pre-processing"
Jul 24 00:44:51.885397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.885289    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90fabf6e-ca71-4235-a900-8635d04b6db3" containerName="payload-processing"
Jul 24 00:44:51.885397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.885295    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="90fabf6e-ca71-4235-a900-8635d04b6db3" containerName="payload-processing"
Jul 24 00:44:51.885397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.885355    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="90fabf6e-ca71-4235-a900-8635d04b6db3" containerName="payload-processing"
Jul 24 00:44:51.885397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.885364    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d5e314-8c87-489a-8f05-211c5abb5310" containerName="maas-api"
Jul 24 00:44:51.885397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.885372    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5bd6ab0-800c-4480-9da1-529a116bb01b" containerName="payload-pre-processing"
Jul 24 00:44:51.888350 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.888329    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:51.891215 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.891192    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-587b8d23-serving-cert\""
Jul 24 00:44:51.900321 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.900293    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"]
Jul 24 00:44:51.939244 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.939196    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvqs\" (UniqueName: \"kubernetes.io/projected/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-kube-api-access-vzvqs\") pod \"maas-api-e2e-mt-587b8d23-745d756458-2sbfq\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:51.939416 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.939252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-maas-api-tls\") pod \"maas-api-e2e-mt-587b8d23-745d756458-2sbfq\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:51.939416 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:51.939337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-openshift-service-ca\") pod \"maas-api-e2e-mt-587b8d23-745d756458-2sbfq\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:52.040764 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.040718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvqs\" (UniqueName: \"kubernetes.io/projected/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-kube-api-access-vzvqs\") pod \"maas-api-e2e-mt-587b8d23-745d756458-2sbfq\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:52.040764 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.040765    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-maas-api-tls\") pod \"maas-api-e2e-mt-587b8d23-745d756458-2sbfq\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:52.041030 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.040795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-openshift-service-ca\") pod \"maas-api-e2e-mt-587b8d23-745d756458-2sbfq\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:52.041030 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:44:52.040929    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-serving-cert: secret "maas-api-e2e-mt-587b8d23-serving-cert" not found
Jul 24 00:44:52.041176 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:44:52.041042    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-maas-api-tls podName:d5ad7377-cf50-40f0-b321-fe5171d9c9f7 nodeName:}" failed. No retries permitted until 2026-07-24 00:44:52.541018758 +0000 UTC m=+1873.108834892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-maas-api-tls") pod "maas-api-e2e-mt-587b8d23-745d756458-2sbfq" (UID: "d5ad7377-cf50-40f0-b321-fe5171d9c9f7") : secret "maas-api-e2e-mt-587b8d23-serving-cert" not found
Jul 24 00:44:52.041654 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.041628    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-openshift-service-ca\") pod \"maas-api-e2e-mt-587b8d23-745d756458-2sbfq\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:52.050026 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.050000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvqs\" (UniqueName: \"kubernetes.io/projected/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-kube-api-access-vzvqs\") pod \"maas-api-e2e-mt-587b8d23-745d756458-2sbfq\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:52.538583 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.538546    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"]
Jul 24 00:44:52.541787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.541769    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:44:52.544321 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.544301    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-587b8d23\""
Jul 24 00:44:52.545171 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.545153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-maas-api-tls\") pod \"maas-api-e2e-mt-587b8d23-745d756458-2sbfq\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:52.547567 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.547536    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-maas-api-tls\") pod \"maas-api-e2e-mt-587b8d23-745d756458-2sbfq\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:52.552840 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.552795    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"]
Jul 24 00:44:52.583858 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.583807    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"]
Jul 24 00:44:52.587080 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.587062    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:44:52.589551 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.589533    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-587b8d23-dockercfg-hj9gc\""
Jul 24 00:44:52.605449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.602903    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"]
Jul 24 00:44:52.645925 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.645886    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p87h5\" (UniqueName: \"kubernetes.io/projected/04265833-831b-49ee-b501-aae649412540-kube-api-access-p87h5\") pod \"payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69\" (UID: \"04265833-831b-49ee-b501-aae649412540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:44:52.645925 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.645923    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528mw\" (UniqueName: \"kubernetes.io/projected/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-kube-api-access-528mw\") pod \"payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7\" (UID: \"3d99632b-bb6c-4c1e-a0cd-3184cbc37028\") " pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:44:52.646179 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.646045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-plugins-config-volume\") pod \"payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7\" (UID: \"3d99632b-bb6c-4c1e-a0cd-3184cbc37028\") " pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:44:52.646179 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.646099    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04265833-831b-49ee-b501-aae649412540-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69\" (UID: \"04265833-831b-49ee-b501-aae649412540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:44:52.746950 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.746911    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p87h5\" (UniqueName: \"kubernetes.io/projected/04265833-831b-49ee-b501-aae649412540-kube-api-access-p87h5\") pod \"payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69\" (UID: \"04265833-831b-49ee-b501-aae649412540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:44:52.746950 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.746951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528mw\" (UniqueName: \"kubernetes.io/projected/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-kube-api-access-528mw\") pod \"payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7\" (UID: \"3d99632b-bb6c-4c1e-a0cd-3184cbc37028\") " pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:44:52.747192 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.747015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-plugins-config-volume\") pod \"payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7\" (UID: \"3d99632b-bb6c-4c1e-a0cd-3184cbc37028\") " pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:44:52.747192 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.747051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04265833-831b-49ee-b501-aae649412540-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69\" (UID: \"04265833-831b-49ee-b501-aae649412540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:44:52.747794 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.747768    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-plugins-config-volume\") pod \"payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7\" (UID: \"3d99632b-bb6c-4c1e-a0cd-3184cbc37028\") " pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:44:52.747942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.747823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04265833-831b-49ee-b501-aae649412540-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69\" (UID: \"04265833-831b-49ee-b501-aae649412540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:44:52.755686 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.755658    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-528mw\" (UniqueName: \"kubernetes.io/projected/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-kube-api-access-528mw\") pod \"payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7\" (UID: \"3d99632b-bb6c-4c1e-a0cd-3184cbc37028\") " pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:44:52.755792 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.755739    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p87h5\" (UniqueName: \"kubernetes.io/projected/04265833-831b-49ee-b501-aae649412540-kube-api-access-p87h5\") pod \"payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69\" (UID: \"04265833-831b-49ee-b501-aae649412540\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:44:52.801803 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.801712    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:52.862572 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.862535    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:44:52.897322 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.897292    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:44:52.952819 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:52.952785    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"]
Jul 24 00:44:52.955823 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:44:52.955791    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ad7377_cf50_40f0_b321_fe5171d9c9f7.slice/crio-fa108088a0812e4d252f4d79d9da9ebab5187e9c8062998c5262c1826bfd1944 WatchSource:0}: Error finding container fa108088a0812e4d252f4d79d9da9ebab5187e9c8062998c5262c1826bfd1944: Status 404 returned error can't find the container with id fa108088a0812e4d252f4d79d9da9ebab5187e9c8062998c5262c1826bfd1944
Jul 24 00:44:53.023432 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:53.023383    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"]
Jul 24 00:44:53.028230 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:44:53.028188    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04265833_831b_49ee_b501_aae649412540.slice/crio-16a5a31f966478ba2a17cd313aa0b7b4905027c29b1e98f477c40dac98bad8f4 WatchSource:0}: Error finding container 16a5a31f966478ba2a17cd313aa0b7b4905027c29b1e98f477c40dac98bad8f4: Status 404 returned error can't find the container with id 16a5a31f966478ba2a17cd313aa0b7b4905027c29b1e98f477c40dac98bad8f4
Jul 24 00:44:53.056145 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:53.056094    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"]
Jul 24 00:44:53.058368 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:44:53.058331    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d99632b_bb6c_4c1e_a0cd_3184cbc37028.slice/crio-bc509f6836f8edeb0113201a3943cff0bd6aff250ee22ba278e82dcacabdd0df WatchSource:0}: Error finding container bc509f6836f8edeb0113201a3943cff0bd6aff250ee22ba278e82dcacabdd0df: Status 404 returned error can't find the container with id bc509f6836f8edeb0113201a3943cff0bd6aff250ee22ba278e82dcacabdd0df
Jul 24 00:44:53.327863 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:53.327753    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69" event={"ID":"04265833-831b-49ee-b501-aae649412540","Type":"ContainerStarted","Data":"9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696"}
Jul 24 00:44:53.327863 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:53.327816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69" event={"ID":"04265833-831b-49ee-b501-aae649412540","Type":"ContainerStarted","Data":"16a5a31f966478ba2a17cd313aa0b7b4905027c29b1e98f477c40dac98bad8f4"}
Jul 24 00:44:53.328075 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:53.327876    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:44:53.329010 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:53.328981    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq" event={"ID":"d5ad7377-cf50-40f0-b321-fe5171d9c9f7","Type":"ContainerStarted","Data":"fa108088a0812e4d252f4d79d9da9ebab5187e9c8062998c5262c1826bfd1944"}
Jul 24 00:44:53.330420 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:53.330396    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7" event={"ID":"3d99632b-bb6c-4c1e-a0cd-3184cbc37028","Type":"ContainerStarted","Data":"3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c"}
Jul 24 00:44:53.330530 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:53.330439    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7" event={"ID":"3d99632b-bb6c-4c1e-a0cd-3184cbc37028","Type":"ContainerStarted","Data":"bc509f6836f8edeb0113201a3943cff0bd6aff250ee22ba278e82dcacabdd0df"}
Jul 24 00:44:53.330530 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:53.330471    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:44:53.342735 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:53.342683    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69" podStartSLOduration=1.342664311 podStartE2EDuration="1.342664311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:44:53.342420826 +0000 UTC m=+1873.910236978" watchObservedRunningTime="2026-07-24 00:44:53.342664311 +0000 UTC m=+1873.910480460"
Jul 24 00:44:53.358783 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:53.358724    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7" podStartSLOduration=1.358709572 podStartE2EDuration="1.358709572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:44:53.356724545 +0000 UTC m=+1873.924540696" watchObservedRunningTime="2026-07-24 00:44:53.358709572 +0000 UTC m=+1873.926525722"
Jul 24 00:44:54.336411 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:54.336324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq" event={"ID":"d5ad7377-cf50-40f0-b321-fe5171d9c9f7","Type":"ContainerStarted","Data":"235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8"}
Jul 24 00:44:54.336899 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:54.336864    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:44:54.352951 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:44:54.352894    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq" podStartSLOduration=2.895593364 podStartE2EDuration="3.352874325s" totalImagesPullingTime="457.280961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:44:54.351858175 +0000 UTC m=+1874.919674323" watchObservedRunningTime="2026-07-24 00:44:54.352874325 +0000 UTC m=+1874.920690475"
Jul 24 00:45:01.353599 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:01.353567    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:45:04.338517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:04.338441    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:45:04.338891 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:04.338521    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:45:13.882806 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:13.882765    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"]
Jul 24 00:45:13.883896 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:13.883615    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq" podUID="d5ad7377-cf50-40f0-b321-fe5171d9c9f7" containerName="maas-api" containerID="cri-o://235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8" gracePeriod=30
Jul 24 00:45:13.941145 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:13.941108    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"]
Jul 24 00:45:13.941407 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:13.941358    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7" podUID="3d99632b-bb6c-4c1e-a0cd-3184cbc37028" containerName="payload-processing" containerID="cri-o://3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c" gracePeriod=30
Jul 24 00:45:13.945939 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:13.945914    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"]
Jul 24 00:45:13.946283 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:13.946254    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69" podUID="04265833-831b-49ee-b501-aae649412540" containerName="payload-pre-processing" containerID="cri-o://9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696" gracePeriod=30
Jul 24 00:45:14.202409 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.202375    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:45:14.242776 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.242752    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:45:14.246631 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.246606    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:45:14.338162 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.338127    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-528mw\" (UniqueName: \"kubernetes.io/projected/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-kube-api-access-528mw\") pod \"3d99632b-bb6c-4c1e-a0cd-3184cbc37028\" (UID: \"3d99632b-bb6c-4c1e-a0cd-3184cbc37028\") "
Jul 24 00:45:14.338361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.338218    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-maas-api-tls\") pod \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") "
Jul 24 00:45:14.338361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.338247    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-openshift-service-ca\") pod \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") "
Jul 24 00:45:14.338361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.338283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04265833-831b-49ee-b501-aae649412540-plugins-config-volume\") pod \"04265833-831b-49ee-b501-aae649412540\" (UID: \"04265833-831b-49ee-b501-aae649412540\") "
Jul 24 00:45:14.338361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.338327    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p87h5\" (UniqueName: \"kubernetes.io/projected/04265833-831b-49ee-b501-aae649412540-kube-api-access-p87h5\") pod \"04265833-831b-49ee-b501-aae649412540\" (UID: \"04265833-831b-49ee-b501-aae649412540\") "
Jul 24 00:45:14.338640 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.338617    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-plugins-config-volume\") pod \"3d99632b-bb6c-4c1e-a0cd-3184cbc37028\" (UID: \"3d99632b-bb6c-4c1e-a0cd-3184cbc37028\") "
Jul 24 00:45:14.338861 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.338705    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5ad7377-cf50-40f0-b321-fe5171d9c9f7" (UID: "d5ad7377-cf50-40f0-b321-fe5171d9c9f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:45:14.338861 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.338718    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04265833-831b-49ee-b501-aae649412540-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04265833-831b-49ee-b501-aae649412540" (UID: "04265833-831b-49ee-b501-aae649412540"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:45:14.339041 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.338921    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d99632b-bb6c-4c1e-a0cd-3184cbc37028" (UID: "3d99632b-bb6c-4c1e-a0cd-3184cbc37028"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:45:14.339111 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.338821    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzvqs\" (UniqueName: \"kubernetes.io/projected/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-kube-api-access-vzvqs\") pod \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\" (UID: \"d5ad7377-cf50-40f0-b321-fe5171d9c9f7\") "
Jul 24 00:45:14.339403 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.339383    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.339499 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.339409    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04265833-831b-49ee-b501-aae649412540-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.339499 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.339424    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.340809 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.340775    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04265833-831b-49ee-b501-aae649412540-kube-api-access-p87h5" (OuterVolumeSpecName: "kube-api-access-p87h5") pod "04265833-831b-49ee-b501-aae649412540" (UID: "04265833-831b-49ee-b501-aae649412540"). InnerVolumeSpecName "kube-api-access-p87h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:45:14.340996 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.340975    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-kube-api-access-vzvqs" (OuterVolumeSpecName: "kube-api-access-vzvqs") pod "d5ad7377-cf50-40f0-b321-fe5171d9c9f7" (UID: "d5ad7377-cf50-40f0-b321-fe5171d9c9f7"). InnerVolumeSpecName "kube-api-access-vzvqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:45:14.341095 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.341000    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-kube-api-access-528mw" (OuterVolumeSpecName: "kube-api-access-528mw") pod "3d99632b-bb6c-4c1e-a0cd-3184cbc37028" (UID: "3d99632b-bb6c-4c1e-a0cd-3184cbc37028"). InnerVolumeSpecName "kube-api-access-528mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:45:14.341095 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.341024    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5ad7377-cf50-40f0-b321-fe5171d9c9f7" (UID: "d5ad7377-cf50-40f0-b321-fe5171d9c9f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:45:14.438427 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.438328    2560 generic.go:356] "Generic (PLEG): container finished" podID="04265833-831b-49ee-b501-aae649412540" containerID="9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696" exitCode=0
Jul 24 00:45:14.438427 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.438394    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"
Jul 24 00:45:14.438427 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.438406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69" event={"ID":"04265833-831b-49ee-b501-aae649412540","Type":"ContainerDied","Data":"9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696"}
Jul 24 00:45:14.438686 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.438468    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69" event={"ID":"04265833-831b-49ee-b501-aae649412540","Type":"ContainerDied","Data":"16a5a31f966478ba2a17cd313aa0b7b4905027c29b1e98f477c40dac98bad8f4"}
Jul 24 00:45:14.438686 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.438500    2560 scope.go:117] "RemoveContainer" containerID="9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696"
Jul 24 00:45:14.440020 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.439866    2560 generic.go:356] "Generic (PLEG): container finished" podID="d5ad7377-cf50-40f0-b321-fe5171d9c9f7" containerID="235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8" exitCode=0
Jul 24 00:45:14.440020 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.439893    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq" event={"ID":"d5ad7377-cf50-40f0-b321-fe5171d9c9f7","Type":"ContainerDied","Data":"235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8"}
Jul 24 00:45:14.440020 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.439936    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"
Jul 24 00:45:14.440020 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.439942    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq" event={"ID":"d5ad7377-cf50-40f0-b321-fe5171d9c9f7","Type":"ContainerDied","Data":"fa108088a0812e4d252f4d79d9da9ebab5187e9c8062998c5262c1826bfd1944"}
Jul 24 00:45:14.440262 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.439873    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p87h5\" (UniqueName: \"kubernetes.io/projected/04265833-831b-49ee-b501-aae649412540-kube-api-access-p87h5\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.440262 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.440109    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzvqs\" (UniqueName: \"kubernetes.io/projected/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-kube-api-access-vzvqs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.440262 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.440205    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-528mw\" (UniqueName: \"kubernetes.io/projected/3d99632b-bb6c-4c1e-a0cd-3184cbc37028-kube-api-access-528mw\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.440262 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.440246    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ad7377-cf50-40f0-b321-fe5171d9c9f7-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:45:14.441651 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.441621    2560 generic.go:356] "Generic (PLEG): container finished" podID="3d99632b-bb6c-4c1e-a0cd-3184cbc37028" containerID="3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c" exitCode=0
Jul 24 00:45:14.441782 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.441672    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"
Jul 24 00:45:14.441882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.441812    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7" event={"ID":"3d99632b-bb6c-4c1e-a0cd-3184cbc37028","Type":"ContainerDied","Data":"3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c"}
Jul 24 00:45:14.441882 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.441863    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7" event={"ID":"3d99632b-bb6c-4c1e-a0cd-3184cbc37028","Type":"ContainerDied","Data":"bc509f6836f8edeb0113201a3943cff0bd6aff250ee22ba278e82dcacabdd0df"}
Jul 24 00:45:14.449322 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.449299    2560 scope.go:117] "RemoveContainer" containerID="9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696"
Jul 24 00:45:14.449618 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:45:14.449594    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696\": container with ID starting with 9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696 not found: ID does not exist" containerID="9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696"
Jul 24 00:45:14.449696 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.449628    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696"} err="failed to get container status \"9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696\": rpc error: code = NotFound desc = could not find container \"9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696\": container with ID starting with 9c936030a65e47d312dbdc2eaf6d43038be6e78a7d78fbfc92ab7a2d667b4696 not found: ID does not exist"
Jul 24 00:45:14.449696 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.449650    2560 scope.go:117] "RemoveContainer" containerID="235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8"
Jul 24 00:45:14.457863 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.457815    2560 scope.go:117] "RemoveContainer" containerID="235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8"
Jul 24 00:45:14.458132 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:45:14.458112    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8\": container with ID starting with 235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8 not found: ID does not exist" containerID="235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8"
Jul 24 00:45:14.458184 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.458138    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8"} err="failed to get container status \"235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8\": rpc error: code = NotFound desc = could not find container \"235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8\": container with ID starting with 235add2bad7554080ad18a06423c95232626644db572c2d5414a9abdd815b1f8 not found: ID does not exist"
Jul 24 00:45:14.458184 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.458154    2560 scope.go:117] "RemoveContainer" containerID="3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c"
Jul 24 00:45:14.463529 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.463506    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"]
Jul 24 00:45:14.467461 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.467443    2560 scope.go:117] "RemoveContainer" containerID="3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c"
Jul 24 00:45:14.467739 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:45:14.467719    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c\": container with ID starting with 3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c not found: ID does not exist" containerID="3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c"
Jul 24 00:45:14.467805 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.467744    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c"} err="failed to get container status \"3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c\": rpc error: code = NotFound desc = could not find container \"3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c\": container with ID starting with 3a07e670c940c8eef578c2df7ebdc7101711fafdcd27ca50e49e3cb883b0dd3c not found: ID does not exist"
Jul 24 00:45:14.468064 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.468046    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-587b8d23-586f5676bd-nxh69"]
Jul 24 00:45:14.477166 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.477143    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"]
Jul 24 00:45:14.482953 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.482928    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-587b8d23-6745bb458b-dcmm7"]
Jul 24 00:45:14.493381 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.493349    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"]
Jul 24 00:45:14.494739 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:14.494718    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-587b8d23-745d756458-2sbfq"]
Jul 24 00:45:15.966704 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:15.966668    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04265833-831b-49ee-b501-aae649412540" path="/var/lib/kubelet/pods/04265833-831b-49ee-b501-aae649412540/volumes"
Jul 24 00:45:15.967086 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:15.967055    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d99632b-bb6c-4c1e-a0cd-3184cbc37028" path="/var/lib/kubelet/pods/3d99632b-bb6c-4c1e-a0cd-3184cbc37028/volumes"
Jul 24 00:45:15.967391 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:15.967378    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5ad7377-cf50-40f0-b321-fe5171d9c9f7" path="/var/lib/kubelet/pods/d5ad7377-cf50-40f0-b321-fe5171d9c9f7/volumes"
Jul 24 00:45:47.173086 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.173052    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"]
Jul 24 00:45:47.173657 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.173420    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5ad7377-cf50-40f0-b321-fe5171d9c9f7" containerName="maas-api"
Jul 24 00:45:47.173657 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.173431    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5ad7377-cf50-40f0-b321-fe5171d9c9f7" containerName="maas-api"
Jul 24 00:45:47.173657 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.173444    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04265833-831b-49ee-b501-aae649412540" containerName="payload-pre-processing"
Jul 24 00:45:47.173657 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.173450    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="04265833-831b-49ee-b501-aae649412540" containerName="payload-pre-processing"
Jul 24 00:45:47.173657 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.173457    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d99632b-bb6c-4c1e-a0cd-3184cbc37028" containerName="payload-processing"
Jul 24 00:45:47.173657 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.173463    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d99632b-bb6c-4c1e-a0cd-3184cbc37028" containerName="payload-processing"
Jul 24 00:45:47.173657 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.173515    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d99632b-bb6c-4c1e-a0cd-3184cbc37028" containerName="payload-processing"
Jul 24 00:45:47.173657 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.173523    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="04265833-831b-49ee-b501-aae649412540" containerName="payload-pre-processing"
Jul 24 00:45:47.173657 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.173529    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5ad7377-cf50-40f0-b321-fe5171d9c9f7" containerName="maas-api"
Jul 24 00:45:47.176463 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.176444    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:47.179471 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.179444    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c4e06e57-serving-cert\""
Jul 24 00:45:47.189375 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.189351    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"]
Jul 24 00:45:47.216814 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.216769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-maas-api-tls\") pod \"maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:47.216995 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.216823    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pqqp\" (UniqueName: \"kubernetes.io/projected/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-kube-api-access-6pqqp\") pod \"maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:47.217045 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.216984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-openshift-service-ca\") pod \"maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:47.317429 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.317399    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pqqp\" (UniqueName: \"kubernetes.io/projected/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-kube-api-access-6pqqp\") pod \"maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:47.317596 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.317460    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-openshift-service-ca\") pod \"maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:47.317596 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.317491    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-maas-api-tls\") pod \"maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:47.318163 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.318142    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-openshift-service-ca\") pod \"maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:47.319966 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.319943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-maas-api-tls\") pod \"maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:47.334802 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.334775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pqqp\" (UniqueName: \"kubernetes.io/projected/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-kube-api-access-6pqqp\") pod \"maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:47.487001 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.486969    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:47.629232 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.629205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"]
Jul 24 00:45:47.632294 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:45:47.632264    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf0ddfc_6b3b_48b1_9ef5_1c1ecc7c3c51.slice/crio-23c4c3e0fac85acc5ef8eab2a4f60645f3bceb6390b82be5d939d436a8f3efe5 WatchSource:0}: Error finding container 23c4c3e0fac85acc5ef8eab2a4f60645f3bceb6390b82be5d939d436a8f3efe5: Status 404 returned error can't find the container with id 23c4c3e0fac85acc5ef8eab2a4f60645f3bceb6390b82be5d939d436a8f3efe5
Jul 24 00:45:47.633699 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.633670    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:45:47.918765 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.918690    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"]
Jul 24 00:45:47.923258 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.923236    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:45:47.925549 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.925524    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c4e06e57\""
Jul 24 00:45:47.934161 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.934139    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"]
Jul 24 00:45:47.958271 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.958236    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"]
Jul 24 00:45:47.961915 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.961884    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:45:47.964436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.964414    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c4e06e57-dockercfg-tb4h8\""
Jul 24 00:45:47.974766 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:47.974739    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"]
Jul 24 00:45:48.024766 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.024730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtqmc\" (UniqueName: \"kubernetes.io/projected/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-kube-api-access-wtqmc\") pod \"payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx\" (UID: \"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:45:48.024947 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.024784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgpcw\" (UniqueName: \"kubernetes.io/projected/fbc0601a-1a51-4844-bf1e-2a3732670962-kube-api-access-dgpcw\") pod \"payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4\" (UID: \"fbc0601a-1a51-4844-bf1e-2a3732670962\") " pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:45:48.025001 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.024965    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx\" (UID: \"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:45:48.025001 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.024992    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc0601a-1a51-4844-bf1e-2a3732670962-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4\" (UID: \"fbc0601a-1a51-4844-bf1e-2a3732670962\") " pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:45:48.125845 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.125806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtqmc\" (UniqueName: \"kubernetes.io/projected/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-kube-api-access-wtqmc\") pod \"payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx\" (UID: \"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:45:48.125975 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.125876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgpcw\" (UniqueName: \"kubernetes.io/projected/fbc0601a-1a51-4844-bf1e-2a3732670962-kube-api-access-dgpcw\") pod \"payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4\" (UID: \"fbc0601a-1a51-4844-bf1e-2a3732670962\") " pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:45:48.125975 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.125946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx\" (UID: \"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:45:48.125975 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.125963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc0601a-1a51-4844-bf1e-2a3732670962-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4\" (UID: \"fbc0601a-1a51-4844-bf1e-2a3732670962\") " pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:45:48.126586 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.126568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc0601a-1a51-4844-bf1e-2a3732670962-plugins-config-volume\") pod \"payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4\" (UID: \"fbc0601a-1a51-4844-bf1e-2a3732670962\") " pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:45:48.126682 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.126663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx\" (UID: \"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:45:48.133871 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.133822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgpcw\" (UniqueName: \"kubernetes.io/projected/fbc0601a-1a51-4844-bf1e-2a3732670962-kube-api-access-dgpcw\") pod \"payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4\" (UID: \"fbc0601a-1a51-4844-bf1e-2a3732670962\") " pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:45:48.134235 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.134214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtqmc\" (UniqueName: \"kubernetes.io/projected/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-kube-api-access-wtqmc\") pod \"payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx\" (UID: \"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:45:48.233728 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.233701    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:45:48.272985 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.272942    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:45:48.401253 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.401148    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"]
Jul 24 00:45:48.457710 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.457685    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"]
Jul 24 00:45:48.460440 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:45:48.460413    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc0601a_1a51_4844_bf1e_2a3732670962.slice/crio-ff65c7e5be749dcaa02f1fa495ba863a4758f123a88ef53be49bcb82c95e2ee7 WatchSource:0}: Error finding container ff65c7e5be749dcaa02f1fa495ba863a4758f123a88ef53be49bcb82c95e2ee7: Status 404 returned error can't find the container with id ff65c7e5be749dcaa02f1fa495ba863a4758f123a88ef53be49bcb82c95e2ee7
Jul 24 00:45:48.602442 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.602402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx" event={"ID":"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5","Type":"ContainerStarted","Data":"fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20"}
Jul 24 00:45:48.602653 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.602476    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx" event={"ID":"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5","Type":"ContainerStarted","Data":"2aa83d06aafda9742f0e1b1ec690e918597450807a1b44ccf37dae4bcf0addc5"}
Jul 24 00:45:48.602653 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.602511    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:45:48.604090 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.604047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv" event={"ID":"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51","Type":"ContainerStarted","Data":"ddf1715d0faf2227b5e225e16add8a4336409a63c943fb25a9bbbe0bda831a90"}
Jul 24 00:45:48.604205 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.604105    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:48.604205 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.604130    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv" event={"ID":"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51","Type":"ContainerStarted","Data":"23c4c3e0fac85acc5ef8eab2a4f60645f3bceb6390b82be5d939d436a8f3efe5"}
Jul 24 00:45:48.605538 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.605516    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4" event={"ID":"fbc0601a-1a51-4844-bf1e-2a3732670962","Type":"ContainerStarted","Data":"657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783"}
Jul 24 00:45:48.605644 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.605562    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4" event={"ID":"fbc0601a-1a51-4844-bf1e-2a3732670962","Type":"ContainerStarted","Data":"ff65c7e5be749dcaa02f1fa495ba863a4758f123a88ef53be49bcb82c95e2ee7"}
Jul 24 00:45:48.605714 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.605663    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:45:48.620009 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.619958    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx" podStartSLOduration=1.619942789 podStartE2EDuration="1.619942789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:45:48.619230353 +0000 UTC m=+1929.187046505" watchObservedRunningTime="2026-07-24 00:45:48.619942789 +0000 UTC m=+1929.187758938"
Jul 24 00:45:48.638307 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.638255    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv" podStartSLOduration=1.231131198 podStartE2EDuration="1.638240352s" totalImagesPullingTime="407.109154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:45:48.636808847 +0000 UTC m=+1929.204625001" watchObservedRunningTime="2026-07-24 00:45:48.638240352 +0000 UTC m=+1929.206056502"
Jul 24 00:45:48.653625 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:48.653567    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4" podStartSLOduration=1.653549082 podStartE2EDuration="1.653549082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:45:48.65273593 +0000 UTC m=+1929.220552079" watchObservedRunningTime="2026-07-24 00:45:48.653549082 +0000 UTC m=+1929.221365223"
Jul 24 00:45:54.615064 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:54.615031    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:45:58.813555 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.813517    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"]
Jul 24 00:45:58.817682 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.817657    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:58.820566 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.820540    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c4e06e57\"/\"default-dockercfg-qs7jq\""
Jul 24 00:45:58.820706 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.820560    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c4e06e57\"/\"kube-root-ca.crt\""
Jul 24 00:45:58.821310 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.821292    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c4e06e57\"/\"openshift-service-ca.crt\""
Jul 24 00:45:58.821996 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.821977    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c4e06e57\"/\"e2e-lifecycle-model-c4e06e57-kserve-self-signed-certs\""
Jul 24 00:45:58.835848 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.835798    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"]
Jul 24 00:45:58.921238 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.921200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthqw\" (UniqueName: \"kubernetes.io/projected/82837a0e-57db-4e79-92db-c9e77f19bc16-kube-api-access-pthqw\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:58.921238 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.921241    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-dshm\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:58.921470 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.921320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-home\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:58.921470 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.921366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-model-cache\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:58.921470 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.921388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-tmp-dir\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:58.921470 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:58.921403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82837a0e-57db-4e79-92db-c9e77f19bc16-tls-certs\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.022008 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.021969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-tmp-dir\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.022008 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.022012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82837a0e-57db-4e79-92db-c9e77f19bc16-tls-certs\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.022215 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.022100    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthqw\" (UniqueName: \"kubernetes.io/projected/82837a0e-57db-4e79-92db-c9e77f19bc16-kube-api-access-pthqw\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.022215 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.022131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-dshm\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.022215 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.022211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-home\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.022403 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.022352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-model-cache\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.022643 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.022615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-home\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.022643 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.022637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-tmp-dir\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.022808 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.022704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-model-cache\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.024415 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.024397    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-dshm\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.024681 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.024662    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82837a0e-57db-4e79-92db-c9e77f19bc16-tls-certs\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.040205 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.040183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthqw\" (UniqueName: \"kubernetes.io/projected/82837a0e-57db-4e79-92db-c9e77f19bc16-kube-api-access-pthqw\") pod \"e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") " pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.127442 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.127358    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:45:59.263566 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.263527    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"]
Jul 24 00:45:59.265782 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:45:59.265755    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82837a0e_57db_4e79_92db_c9e77f19bc16.slice/crio-d8ebfc11c08a1168ec526288268dae1a1fdf2224349e16c212b9f66974719222 WatchSource:0}: Error finding container d8ebfc11c08a1168ec526288268dae1a1fdf2224349e16c212b9f66974719222: Status 404 returned error can't find the container with id d8ebfc11c08a1168ec526288268dae1a1fdf2224349e16c212b9f66974719222
Jul 24 00:45:59.611619 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.611589    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:45:59.611817 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.611716    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:45:59.675550 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.675505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr" event={"ID":"82837a0e-57db-4e79-92db-c9e77f19bc16","Type":"ContainerStarted","Data":"c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0"}
Jul 24 00:45:59.675745 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.675560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr" event={"ID":"82837a0e-57db-4e79-92db-c9e77f19bc16","Type":"ContainerStarted","Data":"d8ebfc11c08a1168ec526288268dae1a1fdf2224349e16c212b9f66974719222"}
Jul 24 00:45:59.697497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:45:59.697446    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr" podStartSLOduration=1.697431528 podStartE2EDuration="1.697431528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:45:59.695002648 +0000 UTC m=+1940.262818797" watchObservedRunningTime="2026-07-24 00:45:59.697431528 +0000 UTC m=+1940.265247679"
Jul 24 00:46:09.128245 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:09.128215    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:46:09.130682 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:09.128268    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:46:09.142222 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:09.142200    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:46:09.730769 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:09.730733    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:46:41.744109 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:41.743910    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"]
Jul 24 00:46:41.744659 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:41.744157    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv" podUID="ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51" containerName="maas-api" containerID="cri-o://ddf1715d0faf2227b5e225e16add8a4336409a63c943fb25a9bbbe0bda831a90" gracePeriod=30
Jul 24 00:46:41.800554 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:41.800515    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"]
Jul 24 00:46:41.800806 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:41.800756    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4" podUID="fbc0601a-1a51-4844-bf1e-2a3732670962" containerName="payload-processing" containerID="cri-o://657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783" gracePeriod=30
Jul 24 00:46:41.807456 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:41.807411    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"]
Jul 24 00:46:41.807654 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:41.807629    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx" podUID="2fc1eb72-9f87-442f-af6a-4d8ad42c75f5" containerName="payload-pre-processing" containerID="cri-o://fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20" gracePeriod=30
Jul 24 00:46:41.869531 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:41.869499    2560 generic.go:356] "Generic (PLEG): container finished" podID="ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51" containerID="ddf1715d0faf2227b5e225e16add8a4336409a63c943fb25a9bbbe0bda831a90" exitCode=0
Jul 24 00:46:41.869694 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:41.869554    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv" event={"ID":"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51","Type":"ContainerDied","Data":"ddf1715d0faf2227b5e225e16add8a4336409a63c943fb25a9bbbe0bda831a90"}
Jul 24 00:46:42.054551 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.054524    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:46:42.098703 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.098670    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-openshift-service-ca\") pod \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") "
Jul 24 00:46:42.098822 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.098744    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-maas-api-tls\") pod \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") "
Jul 24 00:46:42.098822 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.098804    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pqqp\" (UniqueName: \"kubernetes.io/projected/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-kube-api-access-6pqqp\") pod \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\" (UID: \"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51\") "
Jul 24 00:46:42.099206 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.099112    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51" (UID: "ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:46:42.101046 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.101024    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-kube-api-access-6pqqp" (OuterVolumeSpecName: "kube-api-access-6pqqp") pod "ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51" (UID: "ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51"). InnerVolumeSpecName "kube-api-access-6pqqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:46:42.101142 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.101066    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51" (UID: "ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:46:42.105708 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.105687    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:46:42.108668 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.108651    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:46:42.199812 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.199780    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtqmc\" (UniqueName: \"kubernetes.io/projected/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-kube-api-access-wtqmc\") pod \"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5\" (UID: \"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5\") "
Jul 24 00:46:42.200024 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.199822    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc0601a-1a51-4844-bf1e-2a3732670962-plugins-config-volume\") pod \"fbc0601a-1a51-4844-bf1e-2a3732670962\" (UID: \"fbc0601a-1a51-4844-bf1e-2a3732670962\") "
Jul 24 00:46:42.200024 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.199890    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgpcw\" (UniqueName: \"kubernetes.io/projected/fbc0601a-1a51-4844-bf1e-2a3732670962-kube-api-access-dgpcw\") pod \"fbc0601a-1a51-4844-bf1e-2a3732670962\" (UID: \"fbc0601a-1a51-4844-bf1e-2a3732670962\") "
Jul 24 00:46:42.200024 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.199937    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-plugins-config-volume\") pod \"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5\" (UID: \"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5\") "
Jul 24 00:46:42.200198 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.200148    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc0601a-1a51-4844-bf1e-2a3732670962-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fbc0601a-1a51-4844-bf1e-2a3732670962" (UID: "fbc0601a-1a51-4844-bf1e-2a3732670962"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:46:42.200292 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.200274    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:46:42.200348 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.200293    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:46:42.200348 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.200304    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pqqp\" (UniqueName: \"kubernetes.io/projected/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51-kube-api-access-6pqqp\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:46:42.200348 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.200300    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2fc1eb72-9f87-442f-af6a-4d8ad42c75f5" (UID: "2fc1eb72-9f87-442f-af6a-4d8ad42c75f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:46:42.200348 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.200312    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc0601a-1a51-4844-bf1e-2a3732670962-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:46:42.202019 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.201995    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc0601a-1a51-4844-bf1e-2a3732670962-kube-api-access-dgpcw" (OuterVolumeSpecName: "kube-api-access-dgpcw") pod "fbc0601a-1a51-4844-bf1e-2a3732670962" (UID: "fbc0601a-1a51-4844-bf1e-2a3732670962"). InnerVolumeSpecName "kube-api-access-dgpcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:46:42.202112 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.202097    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-kube-api-access-wtqmc" (OuterVolumeSpecName: "kube-api-access-wtqmc") pod "2fc1eb72-9f87-442f-af6a-4d8ad42c75f5" (UID: "2fc1eb72-9f87-442f-af6a-4d8ad42c75f5"). InnerVolumeSpecName "kube-api-access-wtqmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:46:42.301616 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.301517    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtqmc\" (UniqueName: \"kubernetes.io/projected/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-kube-api-access-wtqmc\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:46:42.301616 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.301555    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgpcw\" (UniqueName: \"kubernetes.io/projected/fbc0601a-1a51-4844-bf1e-2a3732670962-kube-api-access-dgpcw\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:46:42.301616 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.301564    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:46:42.875871 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.875819    2560 generic.go:356] "Generic (PLEG): container finished" podID="2fc1eb72-9f87-442f-af6a-4d8ad42c75f5" containerID="fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20" exitCode=0
Jul 24 00:46:42.876332 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.875889    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"
Jul 24 00:46:42.876332 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.875899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx" event={"ID":"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5","Type":"ContainerDied","Data":"fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20"}
Jul 24 00:46:42.876332 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.875954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx" event={"ID":"2fc1eb72-9f87-442f-af6a-4d8ad42c75f5","Type":"ContainerDied","Data":"2aa83d06aafda9742f0e1b1ec690e918597450807a1b44ccf37dae4bcf0addc5"}
Jul 24 00:46:42.876332 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.875981    2560 scope.go:117] "RemoveContainer" containerID="fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20"
Jul 24 00:46:42.877458 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.877355    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"
Jul 24 00:46:42.877458 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.877401    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv" event={"ID":"ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51","Type":"ContainerDied","Data":"23c4c3e0fac85acc5ef8eab2a4f60645f3bceb6390b82be5d939d436a8f3efe5"}
Jul 24 00:46:42.878959 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.878933    2560 generic.go:356] "Generic (PLEG): container finished" podID="fbc0601a-1a51-4844-bf1e-2a3732670962" containerID="657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783" exitCode=0
Jul 24 00:46:42.879091 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.878968    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4" event={"ID":"fbc0601a-1a51-4844-bf1e-2a3732670962","Type":"ContainerDied","Data":"657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783"}
Jul 24 00:46:42.879091 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.879003    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"
Jul 24 00:46:42.879091 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.879006    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4" event={"ID":"fbc0601a-1a51-4844-bf1e-2a3732670962","Type":"ContainerDied","Data":"ff65c7e5be749dcaa02f1fa495ba863a4758f123a88ef53be49bcb82c95e2ee7"}
Jul 24 00:46:42.884978 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.884959    2560 scope.go:117] "RemoveContainer" containerID="fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20"
Jul 24 00:46:42.885276 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:46:42.885247    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20\": container with ID starting with fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20 not found: ID does not exist" containerID="fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20"
Jul 24 00:46:42.885358 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.885287    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20"} err="failed to get container status \"fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20\": rpc error: code = NotFound desc = could not find container \"fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20\": container with ID starting with fe9db0eaa86255505f2d87ad9ff67601a02856eccadc6198704cef391c0eee20 not found: ID does not exist"
Jul 24 00:46:42.885358 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.885313    2560 scope.go:117] "RemoveContainer" containerID="ddf1715d0faf2227b5e225e16add8a4336409a63c943fb25a9bbbe0bda831a90"
Jul 24 00:46:42.898264 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.898243    2560 scope.go:117] "RemoveContainer" containerID="657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783"
Jul 24 00:46:42.905273 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.905245    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"]
Jul 24 00:46:42.906968 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.906934    2560 scope.go:117] "RemoveContainer" containerID="657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783"
Jul 24 00:46:42.907243 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:46:42.907219    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783\": container with ID starting with 657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783 not found: ID does not exist" containerID="657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783"
Jul 24 00:46:42.907315 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.907249    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783"} err="failed to get container status \"657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783\": rpc error: code = NotFound desc = could not find container \"657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783\": container with ID starting with 657271b9228902a092786281cec7e40bde506aef787584b2a7c68d1b108b8783 not found: ID does not exist"
Jul 24 00:46:42.907880 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.907849    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4e06e57-5d9959786d-jsmzx"]
Jul 24 00:46:42.918865 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.918841    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"]
Jul 24 00:46:42.924632 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.924607    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4e06e57-864988fdf4-6crpv"]
Jul 24 00:46:42.933360 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.933338    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"]
Jul 24 00:46:42.937648 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:42.937625    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c4e06e57-69f88b4fd-xtlh4"]
Jul 24 00:46:43.967532 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:43.967497    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fc1eb72-9f87-442f-af6a-4d8ad42c75f5" path="/var/lib/kubelet/pods/2fc1eb72-9f87-442f-af6a-4d8ad42c75f5/volumes"
Jul 24 00:46:43.967927 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:43.967915    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51" path="/var/lib/kubelet/pods/ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51/volumes"
Jul 24 00:46:43.968238 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:43.968224    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbc0601a-1a51-4844-bf1e-2a3732670962" path="/var/lib/kubelet/pods/fbc0601a-1a51-4844-bf1e-2a3732670962/volumes"
Jul 24 00:46:54.672283 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:46:54.672243    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"]
Jul 24 00:47:09.691814 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:09.691766    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr" podUID="82837a0e-57db-4e79-92db-c9e77f19bc16" containerName="main" containerID="cri-o://c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0" gracePeriod=45
Jul 24 00:47:09.720609 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:09.720571    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr" podUID="82837a0e-57db-4e79-92db-c9e77f19bc16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 24 00:47:09.942964 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:09.942904    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:47:10.018017 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.017982    2560 generic.go:356] "Generic (PLEG): container finished" podID="82837a0e-57db-4e79-92db-c9e77f19bc16" containerID="c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0" exitCode=0
Jul 24 00:47:10.018185 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.018036    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"
Jul 24 00:47:10.018185 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.018053    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr" event={"ID":"82837a0e-57db-4e79-92db-c9e77f19bc16","Type":"ContainerDied","Data":"c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0"}
Jul 24 00:47:10.018185 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.018085    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr" event={"ID":"82837a0e-57db-4e79-92db-c9e77f19bc16","Type":"ContainerDied","Data":"d8ebfc11c08a1168ec526288268dae1a1fdf2224349e16c212b9f66974719222"}
Jul 24 00:47:10.018185 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.018101    2560 scope.go:117] "RemoveContainer" containerID="c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0"
Jul 24 00:47:10.026185 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.026165    2560 scope.go:117] "RemoveContainer" containerID="c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0"
Jul 24 00:47:10.026454 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:47:10.026436    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0\": container with ID starting with c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0 not found: ID does not exist" containerID="c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0"
Jul 24 00:47:10.026509 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.026462    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0"} err="failed to get container status \"c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0\": rpc error: code = NotFound desc = could not find container \"c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0\": container with ID starting with c7e9224aac09974d04cdd6fad93727f70d8a9b6afbd446bb6bc556cca5ab3de0 not found: ID does not exist"
Jul 24 00:47:10.042623 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.042596    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-tmp-dir\") pod \"82837a0e-57db-4e79-92db-c9e77f19bc16\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") "
Jul 24 00:47:10.042767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.042653    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-home\") pod \"82837a0e-57db-4e79-92db-c9e77f19bc16\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") "
Jul 24 00:47:10.042767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.042669    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82837a0e-57db-4e79-92db-c9e77f19bc16-tls-certs\") pod \"82837a0e-57db-4e79-92db-c9e77f19bc16\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") "
Jul 24 00:47:10.042767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.042691    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-dshm\") pod \"82837a0e-57db-4e79-92db-c9e77f19bc16\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") "
Jul 24 00:47:10.042767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.042711    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pthqw\" (UniqueName: \"kubernetes.io/projected/82837a0e-57db-4e79-92db-c9e77f19bc16-kube-api-access-pthqw\") pod \"82837a0e-57db-4e79-92db-c9e77f19bc16\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") "
Jul 24 00:47:10.042767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.042763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-model-cache\") pod \"82837a0e-57db-4e79-92db-c9e77f19bc16\" (UID: \"82837a0e-57db-4e79-92db-c9e77f19bc16\") "
Jul 24 00:47:10.043078 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.042877    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82837a0e-57db-4e79-92db-c9e77f19bc16" (UID: "82837a0e-57db-4e79-92db-c9e77f19bc16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:47:10.043140 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.043084    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-home" (OuterVolumeSpecName: "home") pod "82837a0e-57db-4e79-92db-c9e77f19bc16" (UID: "82837a0e-57db-4e79-92db-c9e77f19bc16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:47:10.043140 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.043100    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-tmp-dir\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:47:10.043140 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.043108    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-model-cache" (OuterVolumeSpecName: "model-cache") pod "82837a0e-57db-4e79-92db-c9e77f19bc16" (UID: "82837a0e-57db-4e79-92db-c9e77f19bc16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:47:10.044966 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.044939    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-dshm" (OuterVolumeSpecName: "dshm") pod "82837a0e-57db-4e79-92db-c9e77f19bc16" (UID: "82837a0e-57db-4e79-92db-c9e77f19bc16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:47:10.045079 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.044982    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82837a0e-57db-4e79-92db-c9e77f19bc16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82837a0e-57db-4e79-92db-c9e77f19bc16" (UID: "82837a0e-57db-4e79-92db-c9e77f19bc16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:47:10.045079 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.045009    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82837a0e-57db-4e79-92db-c9e77f19bc16-kube-api-access-pthqw" (OuterVolumeSpecName: "kube-api-access-pthqw") pod "82837a0e-57db-4e79-92db-c9e77f19bc16" (UID: "82837a0e-57db-4e79-92db-c9e77f19bc16"). InnerVolumeSpecName "kube-api-access-pthqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:47:10.143804 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.143769    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-dshm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:47:10.143804 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.143798    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pthqw\" (UniqueName: \"kubernetes.io/projected/82837a0e-57db-4e79-92db-c9e77f19bc16-kube-api-access-pthqw\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:47:10.143804 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.143809    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-model-cache\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:47:10.144080 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.143818    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82837a0e-57db-4e79-92db-c9e77f19bc16-home\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:47:10.144080 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.143826    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82837a0e-57db-4e79-92db-c9e77f19bc16-tls-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:47:10.341752 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.341719    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"]
Jul 24 00:47:10.346379 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:10.346352    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c4e06e57/e2e-lifecycle-model-c4e06e57-kserve-798b9d8d7f-bhqtr"]
Jul 24 00:47:11.966641 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:47:11.966604    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82837a0e-57db-4e79-92db-c9e77f19bc16" path="/var/lib/kubelet/pods/82837a0e-57db-4e79-92db-c9e77f19bc16/volumes"
Jul 24 00:48:36.194787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.194748    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"]
Jul 24 00:48:36.195362 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195263    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fc1eb72-9f87-442f-af6a-4d8ad42c75f5" containerName="payload-pre-processing"
Jul 24 00:48:36.195362 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195278    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fc1eb72-9f87-442f-af6a-4d8ad42c75f5" containerName="payload-pre-processing"
Jul 24 00:48:36.195362 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195306    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc0601a-1a51-4844-bf1e-2a3732670962" containerName="payload-processing"
Jul 24 00:48:36.195362 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195315    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc0601a-1a51-4844-bf1e-2a3732670962" containerName="payload-processing"
Jul 24 00:48:36.195362 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195326    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82837a0e-57db-4e79-92db-c9e77f19bc16" containerName="main"
Jul 24 00:48:36.195362 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195335    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="82837a0e-57db-4e79-92db-c9e77f19bc16" containerName="main"
Jul 24 00:48:36.195362 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195354    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51" containerName="maas-api"
Jul 24 00:48:36.195362 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195362    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51" containerName="maas-api"
Jul 24 00:48:36.195630 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195438    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fc1eb72-9f87-442f-af6a-4d8ad42c75f5" containerName="payload-pre-processing"
Jul 24 00:48:36.195630 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195453    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccf0ddfc-6b3b-48b1-9ef5-1c1ecc7c3c51" containerName="maas-api"
Jul 24 00:48:36.195630 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195464    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="82837a0e-57db-4e79-92db-c9e77f19bc16" containerName="main"
Jul 24 00:48:36.195630 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.195473    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc0601a-1a51-4844-bf1e-2a3732670962" containerName="payload-processing"
Jul 24 00:48:36.198839 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.198808    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:36.201194 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.201165    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ac89bc-serving-cert\""
Jul 24 00:48:36.206969 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.206944    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"]
Jul 24 00:48:36.281625 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.281581    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f516970-426a-4fb7-8c17-b8998fb1f03e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:36.281817 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.281697    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmmz\" (UniqueName: \"kubernetes.io/projected/8f516970-426a-4fb7-8c17-b8998fb1f03e-kube-api-access-dmmmz\") pod \"maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:36.281817 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.281727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f516970-426a-4fb7-8c17-b8998fb1f03e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:36.382868 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.382797    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f516970-426a-4fb7-8c17-b8998fb1f03e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:36.383080 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.382931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmmz\" (UniqueName: \"kubernetes.io/projected/8f516970-426a-4fb7-8c17-b8998fb1f03e-kube-api-access-dmmmz\") pod \"maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:36.383080 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.382964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f516970-426a-4fb7-8c17-b8998fb1f03e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:36.383202 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:48:36.383082    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-serving-cert: secret "maas-api-e2e-inf-a-ac89bc-serving-cert" not found
Jul 24 00:48:36.383202 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:48:36.383149    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f516970-426a-4fb7-8c17-b8998fb1f03e-maas-api-tls podName:8f516970-426a-4fb7-8c17-b8998fb1f03e nodeName:}" failed. No retries permitted until 2026-07-24 00:48:36.883133151 +0000 UTC m=+2097.450949278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f516970-426a-4fb7-8c17-b8998fb1f03e-maas-api-tls") pod "maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt" (UID: "8f516970-426a-4fb7-8c17-b8998fb1f03e") : secret "maas-api-e2e-inf-a-ac89bc-serving-cert" not found
Jul 24 00:48:36.383615 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.383590    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f516970-426a-4fb7-8c17-b8998fb1f03e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:36.391862 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.391819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmmz\" (UniqueName: \"kubernetes.io/projected/8f516970-426a-4fb7-8c17-b8998fb1f03e-kube-api-access-dmmmz\") pod \"maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:36.888545 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.888500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f516970-426a-4fb7-8c17-b8998fb1f03e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:36.891086 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.891065    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f516970-426a-4fb7-8c17-b8998fb1f03e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:36.904422 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.904385    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"]
Jul 24 00:48:36.907997 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.907976    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:48:36.910588 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.910560    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ac89bc-dockercfg-58tg6\""
Jul 24 00:48:36.910753 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.910734    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ac89bc\""
Jul 24 00:48:36.916115 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.916092    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"]
Jul 24 00:48:36.937042 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.937008    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"]
Jul 24 00:48:36.940359 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.940336    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:48:36.951519 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.951496    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"]
Jul 24 00:48:36.989950 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.989913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2428\" (UniqueName: \"kubernetes.io/projected/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-kube-api-access-p2428\") pod \"payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl\" (UID: \"c1dd79b1-37db-4602-8d3f-000c8f8ccae3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:48:36.990111 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:36.989958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl\" (UID: \"c1dd79b1-37db-4602-8d3f-000c8f8ccae3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:48:37.091176 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.091137    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl\" (UID: \"c1dd79b1-37db-4602-8d3f-000c8f8ccae3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:48:37.091348 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.091184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52de133-16be-4127-a89a-057fe0fd1c1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b\" (UID: \"d52de133-16be-4127-a89a-057fe0fd1c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:48:37.091348 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.091233    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdg97\" (UniqueName: \"kubernetes.io/projected/d52de133-16be-4127-a89a-057fe0fd1c1f-kube-api-access-wdg97\") pod \"payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b\" (UID: \"d52de133-16be-4127-a89a-057fe0fd1c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:48:37.091440 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.091358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2428\" (UniqueName: \"kubernetes.io/projected/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-kube-api-access-p2428\") pod \"payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl\" (UID: \"c1dd79b1-37db-4602-8d3f-000c8f8ccae3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:48:37.091890 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.091872    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl\" (UID: \"c1dd79b1-37db-4602-8d3f-000c8f8ccae3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:48:37.099250 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.099224    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2428\" (UniqueName: \"kubernetes.io/projected/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-kube-api-access-p2428\") pod \"payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl\" (UID: \"c1dd79b1-37db-4602-8d3f-000c8f8ccae3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:48:37.111484 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.111450    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:37.192678 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.192646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52de133-16be-4127-a89a-057fe0fd1c1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b\" (UID: \"d52de133-16be-4127-a89a-057fe0fd1c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:48:37.192880 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.192689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdg97\" (UniqueName: \"kubernetes.io/projected/d52de133-16be-4127-a89a-057fe0fd1c1f-kube-api-access-wdg97\") pod \"payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b\" (UID: \"d52de133-16be-4127-a89a-057fe0fd1c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:48:37.193922 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.193867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52de133-16be-4127-a89a-057fe0fd1c1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b\" (UID: \"d52de133-16be-4127-a89a-057fe0fd1c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:48:37.201110 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.201085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdg97\" (UniqueName: \"kubernetes.io/projected/d52de133-16be-4127-a89a-057fe0fd1c1f-kube-api-access-wdg97\") pod \"payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b\" (UID: \"d52de133-16be-4127-a89a-057fe0fd1c1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:48:37.220611 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.220582    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:48:37.250682 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.250655    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"]
Jul 24 00:48:37.250770 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.250724    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:48:37.252954 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:48:37.252926    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f516970_426a_4fb7_8c17_b8998fb1f03e.slice/crio-c178c49f02ba94fbd6f00811f6602b280481c00d66554563d3cf9071f65493f0 WatchSource:0}: Error finding container c178c49f02ba94fbd6f00811f6602b280481c00d66554563d3cf9071f65493f0: Status 404 returned error can't find the container with id c178c49f02ba94fbd6f00811f6602b280481c00d66554563d3cf9071f65493f0
Jul 24 00:48:37.382608 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.382570    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"]
Jul 24 00:48:37.384448 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:48:37.384395    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dd79b1_37db_4602_8d3f_000c8f8ccae3.slice/crio-86ecb5a131850496ad0353b15a3ec2500f143934be7b5c124288b99636aff2ac WatchSource:0}: Error finding container 86ecb5a131850496ad0353b15a3ec2500f143934be7b5c124288b99636aff2ac: Status 404 returned error can't find the container with id 86ecb5a131850496ad0353b15a3ec2500f143934be7b5c124288b99636aff2ac
Jul 24 00:48:37.408617 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.408586    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl" event={"ID":"c1dd79b1-37db-4602-8d3f-000c8f8ccae3","Type":"ContainerStarted","Data":"86ecb5a131850496ad0353b15a3ec2500f143934be7b5c124288b99636aff2ac"}
Jul 24 00:48:37.411703 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.411643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt" event={"ID":"8f516970-426a-4fb7-8c17-b8998fb1f03e","Type":"ContainerStarted","Data":"c178c49f02ba94fbd6f00811f6602b280481c00d66554563d3cf9071f65493f0"}
Jul 24 00:48:37.412736 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:37.412680    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"]
Jul 24 00:48:37.415703 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:48:37.415675    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52de133_16be_4127_a89a_057fe0fd1c1f.slice/crio-f2f05a47b7836681a7407a960a7d88b76509efc58006ab4b428e68a610aed6cd WatchSource:0}: Error finding container f2f05a47b7836681a7407a960a7d88b76509efc58006ab4b428e68a610aed6cd: Status 404 returned error can't find the container with id f2f05a47b7836681a7407a960a7d88b76509efc58006ab4b428e68a610aed6cd
Jul 24 00:48:38.417433 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:38.417326    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl" event={"ID":"c1dd79b1-37db-4602-8d3f-000c8f8ccae3","Type":"ContainerStarted","Data":"ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a"}
Jul 24 00:48:38.417433 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:38.417424    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:48:38.418888 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:38.418860    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt" event={"ID":"8f516970-426a-4fb7-8c17-b8998fb1f03e","Type":"ContainerStarted","Data":"d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6"}
Jul 24 00:48:38.419017 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:38.418975    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:38.420399 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:38.420373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b" event={"ID":"d52de133-16be-4127-a89a-057fe0fd1c1f","Type":"ContainerStarted","Data":"305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b"}
Jul 24 00:48:38.420510 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:38.420417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b" event={"ID":"d52de133-16be-4127-a89a-057fe0fd1c1f","Type":"ContainerStarted","Data":"f2f05a47b7836681a7407a960a7d88b76509efc58006ab4b428e68a610aed6cd"}
Jul 24 00:48:38.420510 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:38.420487    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:48:38.435165 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:38.435101    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl" podStartSLOduration=2.435077791 podStartE2EDuration="2.435077791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:38.432963482 +0000 UTC m=+2099.000779631" watchObservedRunningTime="2026-07-24 00:48:38.435077791 +0000 UTC m=+2099.002893939"
Jul 24 00:48:38.448299 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:38.448247    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b" podStartSLOduration=2.448231156 podStartE2EDuration="2.448231156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:38.447238387 +0000 UTC m=+2099.015054529" watchObservedRunningTime="2026-07-24 00:48:38.448231156 +0000 UTC m=+2099.016047307"
Jul 24 00:48:38.463211 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:38.463157    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt" podStartSLOduration=1.6825799940000001 podStartE2EDuration="2.463138456s" totalImagesPullingTime="780.558462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:38.462270673 +0000 UTC m=+2099.030086822" watchObservedRunningTime="2026-07-24 00:48:38.463138456 +0000 UTC m=+2099.030954608"
Jul 24 00:48:44.207380 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.207181    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"]
Jul 24 00:48:44.211363 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.211339    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:44.214036 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.214010    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e7050a-serving-cert\""
Jul 24 00:48:44.225464 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.225436    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"]
Jul 24 00:48:44.358071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.358038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqnbj\" (UniqueName: \"kubernetes.io/projected/37deace1-899a-4797-ad10-6701adf07adb-kube-api-access-lqnbj\") pod \"maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:44.358071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.358077    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37deace1-899a-4797-ad10-6701adf07adb-maas-api-tls\") pod \"maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:44.358282 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.358131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37deace1-899a-4797-ad10-6701adf07adb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:44.431945 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.431915    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:48:44.459361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.459279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqnbj\" (UniqueName: \"kubernetes.io/projected/37deace1-899a-4797-ad10-6701adf07adb-kube-api-access-lqnbj\") pod \"maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:44.459361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.459331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37deace1-899a-4797-ad10-6701adf07adb-maas-api-tls\") pod \"maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:44.459584 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.459392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37deace1-899a-4797-ad10-6701adf07adb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:44.460295 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.460261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37deace1-899a-4797-ad10-6701adf07adb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:44.461900 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.461870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37deace1-899a-4797-ad10-6701adf07adb-maas-api-tls\") pod \"maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:44.467400 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.467371    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqnbj\" (UniqueName: \"kubernetes.io/projected/37deace1-899a-4797-ad10-6701adf07adb-kube-api-access-lqnbj\") pod \"maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:44.525366 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.525335    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:44.658978 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.658945    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"]
Jul 24 00:48:44.662999 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:48:44.662945    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37deace1_899a_4797_ad10_6701adf07adb.slice/crio-90159788ca68be50a15b1956ae9b244da439b526a1ae1a291461381e8221cddd WatchSource:0}: Error finding container 90159788ca68be50a15b1956ae9b244da439b526a1ae1a291461381e8221cddd: Status 404 returned error can't find the container with id 90159788ca68be50a15b1956ae9b244da439b526a1ae1a291461381e8221cddd
Jul 24 00:48:44.810517 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.810437    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"]
Jul 24 00:48:44.815005 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.814980    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:48:44.817710 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.817689    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e7050a-dockercfg-w2txd\""
Jul 24 00:48:44.817804 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.817764    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e7050a\""
Jul 24 00:48:44.831036 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.831013    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"]
Jul 24 00:48:44.843652 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.843624    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"]
Jul 24 00:48:44.847923 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.847904    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:48:44.858545 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.858519    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"]
Jul 24 00:48:44.963096 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.963057    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdnl\" (UniqueName: \"kubernetes.io/projected/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-kube-api-access-spdnl\") pod \"payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7\" (UID: \"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:48:44.963267 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.963111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdxnf\" (UniqueName: \"kubernetes.io/projected/fdca376c-859e-4c8d-8161-f3113c1c6950-kube-api-access-cdxnf\") pod \"payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f\" (UID: \"fdca376c-859e-4c8d-8161-f3113c1c6950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:48:44.963267 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.963173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdca376c-859e-4c8d-8161-f3113c1c6950-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f\" (UID: \"fdca376c-859e-4c8d-8161-f3113c1c6950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:48:44.963354 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:44.963316    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7\" (UID: \"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:48:45.064847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.064766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdxnf\" (UniqueName: \"kubernetes.io/projected/fdca376c-859e-4c8d-8161-f3113c1c6950-kube-api-access-cdxnf\") pod \"payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f\" (UID: \"fdca376c-859e-4c8d-8161-f3113c1c6950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:48:45.064847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.064811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdca376c-859e-4c8d-8161-f3113c1c6950-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f\" (UID: \"fdca376c-859e-4c8d-8161-f3113c1c6950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:48:45.065023 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.064910    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7\" (UID: \"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:48:45.065023 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.064935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdnl\" (UniqueName: \"kubernetes.io/projected/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-kube-api-access-spdnl\") pod \"payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7\" (UID: \"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:48:45.065752 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.065722    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7\" (UID: \"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:48:45.065849 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.065799    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdca376c-859e-4c8d-8161-f3113c1c6950-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f\" (UID: \"fdca376c-859e-4c8d-8161-f3113c1c6950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:48:45.072241 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.072221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdxnf\" (UniqueName: \"kubernetes.io/projected/fdca376c-859e-4c8d-8161-f3113c1c6950-kube-api-access-cdxnf\") pod \"payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f\" (UID: \"fdca376c-859e-4c8d-8161-f3113c1c6950\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:48:45.072331 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.072304    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdnl\" (UniqueName: \"kubernetes.io/projected/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-kube-api-access-spdnl\") pod \"payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7\" (UID: \"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:48:45.125714 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.125685    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:48:45.160415 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.160382    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:48:45.283295 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.283264    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"]
Jul 24 00:48:45.285683 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:48:45.285651    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d133aa_f20f_4b03_8b09_7ebfe84dd8de.slice/crio-32dee9997fc1dcedb823cd9dd41c66c57390431711c8c689b7374ae621c64fd0 WatchSource:0}: Error finding container 32dee9997fc1dcedb823cd9dd41c66c57390431711c8c689b7374ae621c64fd0: Status 404 returned error can't find the container with id 32dee9997fc1dcedb823cd9dd41c66c57390431711c8c689b7374ae621c64fd0
Jul 24 00:48:45.452404 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.452362    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs" event={"ID":"37deace1-899a-4797-ad10-6701adf07adb","Type":"ContainerStarted","Data":"c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2"}
Jul 24 00:48:45.452580 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.452442    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:45.452580 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.452471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs" event={"ID":"37deace1-899a-4797-ad10-6701adf07adb","Type":"ContainerStarted","Data":"90159788ca68be50a15b1956ae9b244da439b526a1ae1a291461381e8221cddd"}
Jul 24 00:48:45.454249 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.454221    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7" event={"ID":"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de","Type":"ContainerStarted","Data":"128673dfab6eef3595b5c7694309ad1ead94293b411358769461fae12c6a9a47"}
Jul 24 00:48:45.454368 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.454280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7" event={"ID":"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de","Type":"ContainerStarted","Data":"32dee9997fc1dcedb823cd9dd41c66c57390431711c8c689b7374ae621c64fd0"}
Jul 24 00:48:45.454368 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.454325    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:48:45.469229 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.469183    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs" podStartSLOduration=1.152914142 podStartE2EDuration="1.469168281s" totalImagesPullingTime="316.254139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:45.466748378 +0000 UTC m=+2106.034564523" watchObservedRunningTime="2026-07-24 00:48:45.469168281 +0000 UTC m=+2106.036984428"
Jul 24 00:48:45.482864 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.482789    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7" podStartSLOduration=1.482769638 podStartE2EDuration="1.482769638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:45.480176777 +0000 UTC m=+2106.047992928" watchObservedRunningTime="2026-07-24 00:48:45.482769638 +0000 UTC m=+2106.050585788"
Jul 24 00:48:45.515990 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:45.515965    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"]
Jul 24 00:48:45.517775 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:48:45.517740    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdca376c_859e_4c8d_8161_f3113c1c6950.slice/crio-c91dc6b4468c564be4ad95f09e82903dee0e7e9ddf538ce808a05c7cb577bf18 WatchSource:0}: Error finding container c91dc6b4468c564be4ad95f09e82903dee0e7e9ddf538ce808a05c7cb577bf18: Status 404 returned error can't find the container with id c91dc6b4468c564be4ad95f09e82903dee0e7e9ddf538ce808a05c7cb577bf18
Jul 24 00:48:46.467256 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:46.467207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f" event={"ID":"fdca376c-859e-4c8d-8161-f3113c1c6950","Type":"ContainerStarted","Data":"efb29fbeacc19fd1efbde4bedb4db33df659d744d7c8392c5494dd8316bd7ccd"}
Jul 24 00:48:46.467699 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:46.467678    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:48:46.467787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:46.467729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f" event={"ID":"fdca376c-859e-4c8d-8161-f3113c1c6950","Type":"ContainerStarted","Data":"c91dc6b4468c564be4ad95f09e82903dee0e7e9ddf538ce808a05c7cb577bf18"}
Jul 24 00:48:46.484130 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:46.484084    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f" podStartSLOduration=2.484070667 podStartE2EDuration="2.484070667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:46.481879608 +0000 UTC m=+2107.049695756" watchObservedRunningTime="2026-07-24 00:48:46.484070667 +0000 UTC m=+2107.051886816"
Jul 24 00:48:49.426906 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:49.426875    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:48:49.427356 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:49.426948    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:48:51.473548 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:51.473516    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:48:55.094417 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.094376    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"]
Jul 24 00:48:55.098095 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.098072    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.100359 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.100335    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ac89bc\"/\"default-dockercfg-rdr8g\""
Jul 24 00:48:55.100506 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.100360    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ac89bc\"/\"kube-root-ca.crt\""
Jul 24 00:48:55.101245 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.101219    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ac89bc\"/\"openshift-service-ca.crt\""
Jul 24 00:48:55.101454 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.101430    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ac89bc\"/\"test-model-ac89bc-kserve-self-signed-certs\""
Jul 24 00:48:55.107594 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.107565    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"]
Jul 24 00:48:55.149660 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.149625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-dshm\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.149660 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.149662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac8baac-cecf-491b-9b5b-3ac206038363-tls-certs\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.149899 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.149689    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-tmp-dir\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.149899 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.149795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-home\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.149899 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.149849    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-model-cache\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.150025 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.149901    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7n4j\" (UniqueName: \"kubernetes.io/projected/2ac8baac-cecf-491b-9b5b-3ac206038363-kube-api-access-k7n4j\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.251140 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.251101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-dshm\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.251140 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.251140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac8baac-cecf-491b-9b5b-3ac206038363-tls-certs\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.251387 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.251163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-tmp-dir\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.251387 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.251212    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-home\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.251387 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.251227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-model-cache\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.251387 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.251277    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7n4j\" (UniqueName: \"kubernetes.io/projected/2ac8baac-cecf-491b-9b5b-3ac206038363-kube-api-access-k7n4j\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.251653 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.251624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-tmp-dir\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.251760 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.251680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-home\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.251760 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.251686    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-model-cache\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.253494 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.253473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-dshm\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.253633 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.253613    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac8baac-cecf-491b-9b5b-3ac206038363-tls-certs\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.258819 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.258796    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7n4j\" (UniqueName: \"kubernetes.io/projected/2ac8baac-cecf-491b-9b5b-3ac206038363-kube-api-access-k7n4j\") pod \"test-model-ac89bc-kserve-69f46cbfc-s2j4l\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") " pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.409563 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.409477    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:48:55.545397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:55.545363    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"]
Jul 24 00:48:55.548398 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:48:55.548362    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ac8baac_cecf_491b_9b5b_3ac206038363.slice/crio-190a5da0eed451cbdaf06b540c069c2ee679bc83e81e946ba6f7ddeb2ef524f2 WatchSource:0}: Error finding container 190a5da0eed451cbdaf06b540c069c2ee679bc83e81e946ba6f7ddeb2ef524f2: Status 404 returned error can't find the container with id 190a5da0eed451cbdaf06b540c069c2ee679bc83e81e946ba6f7ddeb2ef524f2
Jul 24 00:48:56.469421 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:56.469391    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:48:56.519444 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:56.519401    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l" event={"ID":"2ac8baac-cecf-491b-9b5b-3ac206038363","Type":"ContainerStarted","Data":"76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998"}
Jul 24 00:48:56.519610 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:56.519453    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l" event={"ID":"2ac8baac-cecf-491b-9b5b-3ac206038363","Type":"ContainerStarted","Data":"190a5da0eed451cbdaf06b540c069c2ee679bc83e81e946ba6f7ddeb2ef524f2"}
Jul 24 00:48:56.538809 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:56.538760    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l" podStartSLOduration=1.538744494 podStartE2EDuration="1.538744494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:56.537307863 +0000 UTC m=+2117.105124049" watchObservedRunningTime="2026-07-24 00:48:56.538744494 +0000 UTC m=+2117.106560643"
Jul 24 00:48:58.480568 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:48:58.480535    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:49:05.410220 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:49:05.410183    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:49:05.410690 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:49:05.410234    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:49:05.422884 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:49:05.422858    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:49:05.574161 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:49:05.574126    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:50:03.519213 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:03.519174    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"]
Jul 24 00:50:14.966621 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:14.966579    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"]
Jul 24 00:50:14.967124 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:14.966860    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt" podUID="8f516970-426a-4fb7-8c17-b8998fb1f03e" containerName="maas-api" containerID="cri-o://d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6" gracePeriod=30
Jul 24 00:50:15.020096 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.020058    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"]
Jul 24 00:50:15.020359 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.020332    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b" podUID="d52de133-16be-4127-a89a-057fe0fd1c1f" containerName="payload-processing" containerID="cri-o://305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b" gracePeriod=30
Jul 24 00:50:15.029731 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.029699    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"]
Jul 24 00:50:15.030024 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.029975    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl" podUID="c1dd79b1-37db-4602-8d3f-000c8f8ccae3" containerName="payload-pre-processing" containerID="cri-o://ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a" gracePeriod=30
Jul 24 00:50:15.324152 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.324120    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:50:15.327713 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.327691    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:50:15.412869 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.412818    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:50:15.462644 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.462606    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2428\" (UniqueName: \"kubernetes.io/projected/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-kube-api-access-p2428\") pod \"c1dd79b1-37db-4602-8d3f-000c8f8ccae3\" (UID: \"c1dd79b1-37db-4602-8d3f-000c8f8ccae3\") "
Jul 24 00:50:15.462644 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.462644    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdg97\" (UniqueName: \"kubernetes.io/projected/d52de133-16be-4127-a89a-057fe0fd1c1f-kube-api-access-wdg97\") pod \"d52de133-16be-4127-a89a-057fe0fd1c1f\" (UID: \"d52de133-16be-4127-a89a-057fe0fd1c1f\") "
Jul 24 00:50:15.462944 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.462677    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52de133-16be-4127-a89a-057fe0fd1c1f-plugins-config-volume\") pod \"d52de133-16be-4127-a89a-057fe0fd1c1f\" (UID: \"d52de133-16be-4127-a89a-057fe0fd1c1f\") "
Jul 24 00:50:15.462944 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.462737    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-plugins-config-volume\") pod \"c1dd79b1-37db-4602-8d3f-000c8f8ccae3\" (UID: \"c1dd79b1-37db-4602-8d3f-000c8f8ccae3\") "
Jul 24 00:50:15.463172 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.463143    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1dd79b1-37db-4602-8d3f-000c8f8ccae3" (UID: "c1dd79b1-37db-4602-8d3f-000c8f8ccae3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:50:15.463219 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.463144    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52de133-16be-4127-a89a-057fe0fd1c1f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d52de133-16be-4127-a89a-057fe0fd1c1f" (UID: "d52de133-16be-4127-a89a-057fe0fd1c1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:50:15.464914 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.464893    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-kube-api-access-p2428" (OuterVolumeSpecName: "kube-api-access-p2428") pod "c1dd79b1-37db-4602-8d3f-000c8f8ccae3" (UID: "c1dd79b1-37db-4602-8d3f-000c8f8ccae3"). InnerVolumeSpecName "kube-api-access-p2428". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:50:15.464977 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.464913    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52de133-16be-4127-a89a-057fe0fd1c1f-kube-api-access-wdg97" (OuterVolumeSpecName: "kube-api-access-wdg97") pod "d52de133-16be-4127-a89a-057fe0fd1c1f" (UID: "d52de133-16be-4127-a89a-057fe0fd1c1f"). InnerVolumeSpecName "kube-api-access-wdg97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:50:15.563493 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.563409    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f516970-426a-4fb7-8c17-b8998fb1f03e-maas-api-tls\") pod \"8f516970-426a-4fb7-8c17-b8998fb1f03e\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") "
Jul 24 00:50:15.563493 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.563468    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmmmz\" (UniqueName: \"kubernetes.io/projected/8f516970-426a-4fb7-8c17-b8998fb1f03e-kube-api-access-dmmmz\") pod \"8f516970-426a-4fb7-8c17-b8998fb1f03e\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") "
Jul 24 00:50:15.563727 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.563515    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f516970-426a-4fb7-8c17-b8998fb1f03e-openshift-service-ca\") pod \"8f516970-426a-4fb7-8c17-b8998fb1f03e\" (UID: \"8f516970-426a-4fb7-8c17-b8998fb1f03e\") "
Jul 24 00:50:15.563781 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.563749    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2428\" (UniqueName: \"kubernetes.io/projected/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-kube-api-access-p2428\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:15.563781 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.563771    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdg97\" (UniqueName: \"kubernetes.io/projected/d52de133-16be-4127-a89a-057fe0fd1c1f-kube-api-access-wdg97\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:15.563879 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.563785    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52de133-16be-4127-a89a-057fe0fd1c1f-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:15.563879 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.563799    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd79b1-37db-4602-8d3f-000c8f8ccae3-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:15.563981 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.563955    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f516970-426a-4fb7-8c17-b8998fb1f03e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f516970-426a-4fb7-8c17-b8998fb1f03e" (UID: "8f516970-426a-4fb7-8c17-b8998fb1f03e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:50:15.565573 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.565546    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f516970-426a-4fb7-8c17-b8998fb1f03e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f516970-426a-4fb7-8c17-b8998fb1f03e" (UID: "8f516970-426a-4fb7-8c17-b8998fb1f03e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:50:15.565685 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.565549    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f516970-426a-4fb7-8c17-b8998fb1f03e-kube-api-access-dmmmz" (OuterVolumeSpecName: "kube-api-access-dmmmz") pod "8f516970-426a-4fb7-8c17-b8998fb1f03e" (UID: "8f516970-426a-4fb7-8c17-b8998fb1f03e"). InnerVolumeSpecName "kube-api-access-dmmmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:50:15.664436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.664396    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmmmz\" (UniqueName: \"kubernetes.io/projected/8f516970-426a-4fb7-8c17-b8998fb1f03e-kube-api-access-dmmmz\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:15.664436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.664428    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f516970-426a-4fb7-8c17-b8998fb1f03e-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:15.664436 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.664438    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f516970-426a-4fb7-8c17-b8998fb1f03e-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:15.909564 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.909473    2560 generic.go:356] "Generic (PLEG): container finished" podID="d52de133-16be-4127-a89a-057fe0fd1c1f" containerID="305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b" exitCode=0
Jul 24 00:50:15.909564 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.909527    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"
Jul 24 00:50:15.909770 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.909560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b" event={"ID":"d52de133-16be-4127-a89a-057fe0fd1c1f","Type":"ContainerDied","Data":"305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b"}
Jul 24 00:50:15.909770 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.909609    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b" event={"ID":"d52de133-16be-4127-a89a-057fe0fd1c1f","Type":"ContainerDied","Data":"f2f05a47b7836681a7407a960a7d88b76509efc58006ab4b428e68a610aed6cd"}
Jul 24 00:50:15.909770 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.909635    2560 scope.go:117] "RemoveContainer" containerID="305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b"
Jul 24 00:50:15.911047 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.911022    2560 generic.go:356] "Generic (PLEG): container finished" podID="c1dd79b1-37db-4602-8d3f-000c8f8ccae3" containerID="ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a" exitCode=0
Jul 24 00:50:15.911177 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.911058    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"
Jul 24 00:50:15.911177 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.911097    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl" event={"ID":"c1dd79b1-37db-4602-8d3f-000c8f8ccae3","Type":"ContainerDied","Data":"ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a"}
Jul 24 00:50:15.911177 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.911139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl" event={"ID":"c1dd79b1-37db-4602-8d3f-000c8f8ccae3","Type":"ContainerDied","Data":"86ecb5a131850496ad0353b15a3ec2500f143934be7b5c124288b99636aff2ac"}
Jul 24 00:50:15.912565 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.912542    2560 generic.go:356] "Generic (PLEG): container finished" podID="8f516970-426a-4fb7-8c17-b8998fb1f03e" containerID="d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6" exitCode=0
Jul 24 00:50:15.912692 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.912577    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt" event={"ID":"8f516970-426a-4fb7-8c17-b8998fb1f03e","Type":"ContainerDied","Data":"d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6"}
Jul 24 00:50:15.912692 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.912605    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"
Jul 24 00:50:15.912692 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.912614    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt" event={"ID":"8f516970-426a-4fb7-8c17-b8998fb1f03e","Type":"ContainerDied","Data":"c178c49f02ba94fbd6f00811f6602b280481c00d66554563d3cf9071f65493f0"}
Jul 24 00:50:15.920352 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.920335    2560 scope.go:117] "RemoveContainer" containerID="305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b"
Jul 24 00:50:15.920707 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:50:15.920679    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b\": container with ID starting with 305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b not found: ID does not exist" containerID="305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b"
Jul 24 00:50:15.920794 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.920719    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b"} err="failed to get container status \"305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b\": rpc error: code = NotFound desc = could not find container \"305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b\": container with ID starting with 305dfe7aa87853d4db3db252e2a53923a2adcb7decaa53559684a05f23b4855b not found: ID does not exist"
Jul 24 00:50:15.920794 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.920742    2560 scope.go:117] "RemoveContainer" containerID="ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a"
Jul 24 00:50:15.930100 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.930079    2560 scope.go:117] "RemoveContainer" containerID="ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a"
Jul 24 00:50:15.930379 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:50:15.930362    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a\": container with ID starting with ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a not found: ID does not exist" containerID="ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a"
Jul 24 00:50:15.930440 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.930391    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a"} err="failed to get container status \"ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a\": rpc error: code = NotFound desc = could not find container \"ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a\": container with ID starting with ac99c3ea5ec03be8fa87556a9cce09c3fae1bb9dd9cce2a00b31882853c9ee0a not found: ID does not exist"
Jul 24 00:50:15.930440 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.930409    2560 scope.go:117] "RemoveContainer" containerID="d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6"
Jul 24 00:50:15.939080 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.939057    2560 scope.go:117] "RemoveContainer" containerID="d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6"
Jul 24 00:50:15.939162 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.939061    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"]
Jul 24 00:50:15.939778 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:50:15.939729    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6\": container with ID starting with d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6 not found: ID does not exist" containerID="d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6"
Jul 24 00:50:15.939896 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.939770    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6"} err="failed to get container status \"d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6\": rpc error: code = NotFound desc = could not find container \"d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6\": container with ID starting with d7055ee6cc5cf46285f0e76f33dc82d29329556d8c357d044b132d392a235fb6 not found: ID does not exist"
Jul 24 00:50:15.941656 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.941630    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac89bc-9c6447cb6-stm5b"]
Jul 24 00:50:15.951718 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.951692    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"]
Jul 24 00:50:15.955757 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.955734    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac89bc-d596ddc44-hb5gl"]
Jul 24 00:50:15.974983 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.974940    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1dd79b1-37db-4602-8d3f-000c8f8ccae3" path="/var/lib/kubelet/pods/c1dd79b1-37db-4602-8d3f-000c8f8ccae3/volumes"
Jul 24 00:50:15.975463 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.975444    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52de133-16be-4127-a89a-057fe0fd1c1f" path="/var/lib/kubelet/pods/d52de133-16be-4127-a89a-057fe0fd1c1f/volumes"
Jul 24 00:50:15.975802 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.975785    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"]
Jul 24 00:50:15.976768 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:15.976749    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac89bc-666fdbffd7-76krt"]
Jul 24 00:50:17.968593 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:17.968556    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f516970-426a-4fb7-8c17-b8998fb1f03e" path="/var/lib/kubelet/pods/8f516970-426a-4fb7-8c17-b8998fb1f03e/volumes"
Jul 24 00:50:18.539099 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.539053    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l" podUID="2ac8baac-cecf-491b-9b5b-3ac206038363" containerName="main" containerID="cri-o://76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998" gracePeriod=45
Jul 24 00:50:18.788920 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.788890    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:50:18.894893 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.894753    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-dshm\") pod \"2ac8baac-cecf-491b-9b5b-3ac206038363\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") "
Jul 24 00:50:18.894893 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.894811    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac8baac-cecf-491b-9b5b-3ac206038363-tls-certs\") pod \"2ac8baac-cecf-491b-9b5b-3ac206038363\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") "
Jul 24 00:50:18.894893 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.894886    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-home\") pod \"2ac8baac-cecf-491b-9b5b-3ac206038363\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") "
Jul 24 00:50:18.895141 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.894922    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-tmp-dir\") pod \"2ac8baac-cecf-491b-9b5b-3ac206038363\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") "
Jul 24 00:50:18.895141 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.894985    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-model-cache\") pod \"2ac8baac-cecf-491b-9b5b-3ac206038363\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") "
Jul 24 00:50:18.895141 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.895028    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7n4j\" (UniqueName: \"kubernetes.io/projected/2ac8baac-cecf-491b-9b5b-3ac206038363-kube-api-access-k7n4j\") pod \"2ac8baac-cecf-491b-9b5b-3ac206038363\" (UID: \"2ac8baac-cecf-491b-9b5b-3ac206038363\") "
Jul 24 00:50:18.895306 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.895230    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ac8baac-cecf-491b-9b5b-3ac206038363" (UID: "2ac8baac-cecf-491b-9b5b-3ac206038363"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:50:18.895306 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.895238    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-home" (OuterVolumeSpecName: "home") pod "2ac8baac-cecf-491b-9b5b-3ac206038363" (UID: "2ac8baac-cecf-491b-9b5b-3ac206038363"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:50:18.895306 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.895266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ac8baac-cecf-491b-9b5b-3ac206038363" (UID: "2ac8baac-cecf-491b-9b5b-3ac206038363"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:50:18.897143 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.897119    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac8baac-cecf-491b-9b5b-3ac206038363-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ac8baac-cecf-491b-9b5b-3ac206038363" (UID: "2ac8baac-cecf-491b-9b5b-3ac206038363"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:50:18.897274 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.897148    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-dshm" (OuterVolumeSpecName: "dshm") pod "2ac8baac-cecf-491b-9b5b-3ac206038363" (UID: "2ac8baac-cecf-491b-9b5b-3ac206038363"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:50:18.897274 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.897154    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac8baac-cecf-491b-9b5b-3ac206038363-kube-api-access-k7n4j" (OuterVolumeSpecName: "kube-api-access-k7n4j") pod "2ac8baac-cecf-491b-9b5b-3ac206038363" (UID: "2ac8baac-cecf-491b-9b5b-3ac206038363"). InnerVolumeSpecName "kube-api-access-k7n4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:50:18.929210 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.929177    2560 generic.go:356] "Generic (PLEG): container finished" podID="2ac8baac-cecf-491b-9b5b-3ac206038363" containerID="76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998" exitCode=0
Jul 24 00:50:18.929333 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.929228    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"
Jul 24 00:50:18.929333 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.929259    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l" event={"ID":"2ac8baac-cecf-491b-9b5b-3ac206038363","Type":"ContainerDied","Data":"76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998"}
Jul 24 00:50:18.929333 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.929302    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l" event={"ID":"2ac8baac-cecf-491b-9b5b-3ac206038363","Type":"ContainerDied","Data":"190a5da0eed451cbdaf06b540c069c2ee679bc83e81e946ba6f7ddeb2ef524f2"}
Jul 24 00:50:18.929333 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.929320    2560 scope.go:117] "RemoveContainer" containerID="76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998"
Jul 24 00:50:18.938787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.938767    2560 scope.go:117] "RemoveContainer" containerID="76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998"
Jul 24 00:50:18.939080 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:50:18.939062    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998\": container with ID starting with 76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998 not found: ID does not exist" containerID="76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998"
Jul 24 00:50:18.939137 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.939091    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998"} err="failed to get container status \"76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998\": rpc error: code = NotFound desc = could not find container \"76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998\": container with ID starting with 76d729f0b38ac1ad2ee26bcaf49bdb452463a8c1852e027224cb8e47a0c3c998 not found: ID does not exist"
Jul 24 00:50:18.951522 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.951493    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"]
Jul 24 00:50:18.956565 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.956541    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ac89bc/test-model-ac89bc-kserve-69f46cbfc-s2j4l"]
Jul 24 00:50:18.996459 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.996436    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-dshm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:18.996459 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.996458    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac8baac-cecf-491b-9b5b-3ac206038363-tls-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:18.996824 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.996466    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-home\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:18.996824 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.996474    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-tmp-dir\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:18.996824 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.996482    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac8baac-cecf-491b-9b5b-3ac206038363-model-cache\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:18.996824 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:18.996491    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7n4j\" (UniqueName: \"kubernetes.io/projected/2ac8baac-cecf-491b-9b5b-3ac206038363-kube-api-access-k7n4j\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:19.966581 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:19.966546    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ac8baac-cecf-491b-9b5b-3ac206038363" path="/var/lib/kubelet/pods/2ac8baac-cecf-491b-9b5b-3ac206038363/volumes"
Jul 24 00:50:26.342210 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342165    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"]
Jul 24 00:50:26.342613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342546    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dd79b1-37db-4602-8d3f-000c8f8ccae3" containerName="payload-pre-processing"
Jul 24 00:50:26.342613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342558    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dd79b1-37db-4602-8d3f-000c8f8ccae3" containerName="payload-pre-processing"
Jul 24 00:50:26.342613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342571    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f516970-426a-4fb7-8c17-b8998fb1f03e" containerName="maas-api"
Jul 24 00:50:26.342613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342577    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f516970-426a-4fb7-8c17-b8998fb1f03e" containerName="maas-api"
Jul 24 00:50:26.342613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342590    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52de133-16be-4127-a89a-057fe0fd1c1f" containerName="payload-processing"
Jul 24 00:50:26.342613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342596    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52de133-16be-4127-a89a-057fe0fd1c1f" containerName="payload-processing"
Jul 24 00:50:26.342613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342604    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac8baac-cecf-491b-9b5b-3ac206038363" containerName="main"
Jul 24 00:50:26.342613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342609    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac8baac-cecf-491b-9b5b-3ac206038363" containerName="main"
Jul 24 00:50:26.342889 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342664    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac8baac-cecf-491b-9b5b-3ac206038363" containerName="main"
Jul 24 00:50:26.342889 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342674    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52de133-16be-4127-a89a-057fe0fd1c1f" containerName="payload-processing"
Jul 24 00:50:26.342889 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342681    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1dd79b1-37db-4602-8d3f-000c8f8ccae3" containerName="payload-pre-processing"
Jul 24 00:50:26.342889 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.342687    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f516970-426a-4fb7-8c17-b8998fb1f03e" containerName="maas-api"
Jul 24 00:50:26.347124 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.347101    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"
Jul 24 00:50:26.349386 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.349348    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\""
Jul 24 00:50:26.353978 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.353883    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"]
Jul 24 00:50:26.357123 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.357087    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6806b69b-d571-408d-8e00-ff75deef3708-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd\" (UID: \"6806b69b-d571-408d-8e00-ff75deef3708\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"
Jul 24 00:50:26.458546 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.458504    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6806b69b-d571-408d-8e00-ff75deef3708-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd\" (UID: \"6806b69b-d571-408d-8e00-ff75deef3708\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"
Jul 24 00:50:26.459194 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.459165    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6806b69b-d571-408d-8e00-ff75deef3708-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd\" (UID: \"6806b69b-d571-408d-8e00-ff75deef3708\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"
Jul 24 00:50:26.661648 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.661296    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"
Jul 24 00:50:26.798250 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.798223    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"]
Jul 24 00:50:26.800797 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:50:26.800766    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6806b69b_d571_408d_8e00_ff75deef3708.slice/crio-a0f8b43f24a7cca9841f0576b3d496e584c1b6d9d1f97db5a14cce517989c82f WatchSource:0}: Error finding container a0f8b43f24a7cca9841f0576b3d496e584c1b6d9d1f97db5a14cce517989c82f: Status 404 returned error can't find the container with id a0f8b43f24a7cca9841f0576b3d496e584c1b6d9d1f97db5a14cce517989c82f
Jul 24 00:50:26.970902 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:26.970866    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd" event={"ID":"6806b69b-d571-408d-8e00-ff75deef3708","Type":"ContainerStarted","Data":"a0f8b43f24a7cca9841f0576b3d496e584c1b6d9d1f97db5a14cce517989c82f"}
Jul 24 00:50:29.988023 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:29.987986    2560 generic.go:356] "Generic (PLEG): container finished" podID="6806b69b-d571-408d-8e00-ff75deef3708" containerID="6257134ccd2942113474bd4f50c9b9e669d8c342994803d5be0aeda2c684ef9f" exitCode=0
Jul 24 00:50:29.988430 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:29.988040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd" event={"ID":"6806b69b-d571-408d-8e00-ff75deef3708","Type":"ContainerDied","Data":"6257134ccd2942113474bd4f50c9b9e669d8c342994803d5be0aeda2c684ef9f"}
Jul 24 00:50:31.132918 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:31.132891    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"
Jul 24 00:50:31.198999 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:31.198964    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6806b69b-d571-408d-8e00-ff75deef3708-maas-api-service-ca\") pod \"6806b69b-d571-408d-8e00-ff75deef3708\" (UID: \"6806b69b-d571-408d-8e00-ff75deef3708\") "
Jul 24 00:50:31.199343 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:31.199319    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6806b69b-d571-408d-8e00-ff75deef3708-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6806b69b-d571-408d-8e00-ff75deef3708" (UID: "6806b69b-d571-408d-8e00-ff75deef3708"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:50:31.299787 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:31.299697    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6806b69b-d571-408d-8e00-ff75deef3708-maas-api-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:31.998008 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:31.997975    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd" event={"ID":"6806b69b-d571-408d-8e00-ff75deef3708","Type":"ContainerDied","Data":"a0f8b43f24a7cca9841f0576b3d496e584c1b6d9d1f97db5a14cce517989c82f"}
Jul 24 00:50:31.998155 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:31.998026    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0f8b43f24a7cca9841f0576b3d496e584c1b6d9d1f97db5a14cce517989c82f"
Jul 24 00:50:31.998155 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:31.997987    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"
Jul 24 00:50:36.781705 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:36.781668    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"]
Jul 24 00:50:36.782255 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:36.781942    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs" podUID="37deace1-899a-4797-ad10-6701adf07adb" containerName="maas-api" containerID="cri-o://c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2" gracePeriod=30
Jul 24 00:50:36.839468 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:36.839431    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"]
Jul 24 00:50:36.839739 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:36.839702    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f" podUID="fdca376c-859e-4c8d-8161-f3113c1c6950" containerName="payload-processing" containerID="cri-o://efb29fbeacc19fd1efbde4bedb4db33df659d744d7c8392c5494dd8316bd7ccd" gracePeriod=30
Jul 24 00:50:36.846979 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:36.846939    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"]
Jul 24 00:50:36.847460 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:36.847428    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7" podUID="b1d133aa-f20f-4b03-8b09-7ebfe84dd8de" containerName="payload-pre-processing" containerID="cri-o://128673dfab6eef3595b5c7694309ad1ead94293b411358769461fae12c6a9a47" gracePeriod=30
Jul 24 00:50:37.024878 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.024821    2560 generic.go:356] "Generic (PLEG): container finished" podID="b1d133aa-f20f-4b03-8b09-7ebfe84dd8de" containerID="128673dfab6eef3595b5c7694309ad1ead94293b411358769461fae12c6a9a47" exitCode=0
Jul 24 00:50:37.024878 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.024862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7" event={"ID":"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de","Type":"ContainerDied","Data":"128673dfab6eef3595b5c7694309ad1ead94293b411358769461fae12c6a9a47"}
Jul 24 00:50:37.026662 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.026633    2560 generic.go:356] "Generic (PLEG): container finished" podID="fdca376c-859e-4c8d-8161-f3113c1c6950" containerID="efb29fbeacc19fd1efbde4bedb4db33df659d744d7c8392c5494dd8316bd7ccd" exitCode=0
Jul 24 00:50:37.026824 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.026688    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f" event={"ID":"fdca376c-859e-4c8d-8161-f3113c1c6950","Type":"ContainerDied","Data":"efb29fbeacc19fd1efbde4bedb4db33df659d744d7c8392c5494dd8316bd7ccd"}
Jul 24 00:50:37.145401 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.145364    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:50:37.148567 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.148542    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:50:37.242680 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.242654    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:50:37.253720 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.253697    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdca376c-859e-4c8d-8161-f3113c1c6950-plugins-config-volume\") pod \"fdca376c-859e-4c8d-8161-f3113c1c6950\" (UID: \"fdca376c-859e-4c8d-8161-f3113c1c6950\") "
Jul 24 00:50:37.253874 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.253737    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-plugins-config-volume\") pod \"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de\" (UID: \"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de\") "
Jul 24 00:50:37.253874 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.253765    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spdnl\" (UniqueName: \"kubernetes.io/projected/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-kube-api-access-spdnl\") pod \"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de\" (UID: \"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de\") "
Jul 24 00:50:37.253874 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.253789    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdxnf\" (UniqueName: \"kubernetes.io/projected/fdca376c-859e-4c8d-8161-f3113c1c6950-kube-api-access-cdxnf\") pod \"fdca376c-859e-4c8d-8161-f3113c1c6950\" (UID: \"fdca376c-859e-4c8d-8161-f3113c1c6950\") "
Jul 24 00:50:37.254166 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.254143    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdca376c-859e-4c8d-8161-f3113c1c6950-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdca376c-859e-4c8d-8161-f3113c1c6950" (UID: "fdca376c-859e-4c8d-8161-f3113c1c6950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:50:37.254229 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.254182    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1d133aa-f20f-4b03-8b09-7ebfe84dd8de" (UID: "b1d133aa-f20f-4b03-8b09-7ebfe84dd8de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:50:37.255870 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.255848    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdca376c-859e-4c8d-8161-f3113c1c6950-kube-api-access-cdxnf" (OuterVolumeSpecName: "kube-api-access-cdxnf") pod "fdca376c-859e-4c8d-8161-f3113c1c6950" (UID: "fdca376c-859e-4c8d-8161-f3113c1c6950"). InnerVolumeSpecName "kube-api-access-cdxnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:50:37.255870 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.255858    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-kube-api-access-spdnl" (OuterVolumeSpecName: "kube-api-access-spdnl") pod "b1d133aa-f20f-4b03-8b09-7ebfe84dd8de" (UID: "b1d133aa-f20f-4b03-8b09-7ebfe84dd8de"). InnerVolumeSpecName "kube-api-access-spdnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:50:37.354718 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.354618    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqnbj\" (UniqueName: \"kubernetes.io/projected/37deace1-899a-4797-ad10-6701adf07adb-kube-api-access-lqnbj\") pod \"37deace1-899a-4797-ad10-6701adf07adb\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") "
Jul 24 00:50:37.354718 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.354665    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37deace1-899a-4797-ad10-6701adf07adb-maas-api-tls\") pod \"37deace1-899a-4797-ad10-6701adf07adb\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") "
Jul 24 00:50:37.354993 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.354743    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37deace1-899a-4797-ad10-6701adf07adb-openshift-service-ca\") pod \"37deace1-899a-4797-ad10-6701adf07adb\" (UID: \"37deace1-899a-4797-ad10-6701adf07adb\") "
Jul 24 00:50:37.355053 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.355004    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:37.355053 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.355024    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spdnl\" (UniqueName: \"kubernetes.io/projected/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de-kube-api-access-spdnl\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:37.355053 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.355043    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdxnf\" (UniqueName: \"kubernetes.io/projected/fdca376c-859e-4c8d-8161-f3113c1c6950-kube-api-access-cdxnf\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:37.355164 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.355058    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdca376c-859e-4c8d-8161-f3113c1c6950-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:37.355164 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.355146    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37deace1-899a-4797-ad10-6701adf07adb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37deace1-899a-4797-ad10-6701adf07adb" (UID: "37deace1-899a-4797-ad10-6701adf07adb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:50:37.356841 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.356804    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37deace1-899a-4797-ad10-6701adf07adb-kube-api-access-lqnbj" (OuterVolumeSpecName: "kube-api-access-lqnbj") pod "37deace1-899a-4797-ad10-6701adf07adb" (UID: "37deace1-899a-4797-ad10-6701adf07adb"). InnerVolumeSpecName "kube-api-access-lqnbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:50:37.356926 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.356855    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37deace1-899a-4797-ad10-6701adf07adb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37deace1-899a-4797-ad10-6701adf07adb" (UID: "37deace1-899a-4797-ad10-6701adf07adb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:50:37.456449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.456405    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqnbj\" (UniqueName: \"kubernetes.io/projected/37deace1-899a-4797-ad10-6701adf07adb-kube-api-access-lqnbj\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:37.456449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.456442    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37deace1-899a-4797-ad10-6701adf07adb-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:37.456449 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:37.456456    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37deace1-899a-4797-ad10-6701adf07adb-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:50:38.032597 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.032568    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"
Jul 24 00:50:38.033101 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.032561    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7" event={"ID":"b1d133aa-f20f-4b03-8b09-7ebfe84dd8de","Type":"ContainerDied","Data":"32dee9997fc1dcedb823cd9dd41c66c57390431711c8c689b7374ae621c64fd0"}
Jul 24 00:50:38.033101 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.032739    2560 scope.go:117] "RemoveContainer" containerID="128673dfab6eef3595b5c7694309ad1ead94293b411358769461fae12c6a9a47"
Jul 24 00:50:38.034245 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.034225    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"
Jul 24 00:50:38.034328 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.034238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f" event={"ID":"fdca376c-859e-4c8d-8161-f3113c1c6950","Type":"ContainerDied","Data":"c91dc6b4468c564be4ad95f09e82903dee0e7e9ddf538ce808a05c7cb577bf18"}
Jul 24 00:50:38.035911 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.035881    2560 generic.go:356] "Generic (PLEG): container finished" podID="37deace1-899a-4797-ad10-6701adf07adb" containerID="c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2" exitCode=0
Jul 24 00:50:38.036048 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.035962    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"
Jul 24 00:50:38.036048 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.035951    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs" event={"ID":"37deace1-899a-4797-ad10-6701adf07adb","Type":"ContainerDied","Data":"c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2"}
Jul 24 00:50:38.036048 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.036016    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs" event={"ID":"37deace1-899a-4797-ad10-6701adf07adb","Type":"ContainerDied","Data":"90159788ca68be50a15b1956ae9b244da439b526a1ae1a291461381e8221cddd"}
Jul 24 00:50:38.065472 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.065449    2560 scope.go:117] "RemoveContainer" containerID="efb29fbeacc19fd1efbde4bedb4db33df659d744d7c8392c5494dd8316bd7ccd"
Jul 24 00:50:38.072404 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.072374    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"]
Jul 24 00:50:38.074056 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.074036    2560 scope.go:117] "RemoveContainer" containerID="c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2"
Jul 24 00:50:38.075543 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.075525    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e7050a-c77b4b69b-dmfj7"]
Jul 24 00:50:38.082012 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.081992    2560 scope.go:117] "RemoveContainer" containerID="c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2"
Jul 24 00:50:38.082292 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:50:38.082273    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2\": container with ID starting with c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2 not found: ID does not exist" containerID="c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2"
Jul 24 00:50:38.082342 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.082302    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2"} err="failed to get container status \"c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2\": rpc error: code = NotFound desc = could not find container \"c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2\": container with ID starting with c8e1afba23935d820489116d2016155f0ef149b09809be3347260058a4454bc2 not found: ID does not exist"
Jul 24 00:50:38.085466 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.085441    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"]
Jul 24 00:50:38.090672 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.090642    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e7050a-5b8467bd4d-ch7qs"]
Jul 24 00:50:38.100745 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.100720    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"]
Jul 24 00:50:38.106875 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:38.106853    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e7050a-9f8cb7868-x8q7f"]
Jul 24 00:50:39.967304 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:39.967269    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37deace1-899a-4797-ad10-6701adf07adb" path="/var/lib/kubelet/pods/37deace1-899a-4797-ad10-6701adf07adb/volumes"
Jul 24 00:50:39.967642 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:39.967623    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1d133aa-f20f-4b03-8b09-7ebfe84dd8de" path="/var/lib/kubelet/pods/b1d133aa-f20f-4b03-8b09-7ebfe84dd8de/volumes"
Jul 24 00:50:39.967952 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:39.967939    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdca376c-859e-4c8d-8161-f3113c1c6950" path="/var/lib/kubelet/pods/fdca376c-859e-4c8d-8161-f3113c1c6950/volumes"
Jul 24 00:50:41.417582 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:41.417545    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"]
Jul 24 00:50:41.420772 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:41.420746    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e7050a-6b8xd"]
Jul 24 00:50:41.967051 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:41.967017    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6806b69b-d571-408d-8e00-ff75deef3708" path="/var/lib/kubelet/pods/6806b69b-d571-408d-8e00-ff75deef3708/volumes"
Jul 24 00:50:51.907125 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907079    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"]
Jul 24 00:50:51.907604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907465    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdca376c-859e-4c8d-8161-f3113c1c6950" containerName="payload-processing"
Jul 24 00:50:51.907604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907479    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdca376c-859e-4c8d-8161-f3113c1c6950" containerName="payload-processing"
Jul 24 00:50:51.907604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907490    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6806b69b-d571-408d-8e00-ff75deef3708" containerName="revoke-keys"
Jul 24 00:50:51.907604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907497    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6806b69b-d571-408d-8e00-ff75deef3708" containerName="revoke-keys"
Jul 24 00:50:51.907604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907511    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37deace1-899a-4797-ad10-6701adf07adb" containerName="maas-api"
Jul 24 00:50:51.907604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907517    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="37deace1-899a-4797-ad10-6701adf07adb" containerName="maas-api"
Jul 24 00:50:51.907604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907527    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d133aa-f20f-4b03-8b09-7ebfe84dd8de" containerName="payload-pre-processing"
Jul 24 00:50:51.907604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907533    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d133aa-f20f-4b03-8b09-7ebfe84dd8de" containerName="payload-pre-processing"
Jul 24 00:50:51.907604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907589    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdca376c-859e-4c8d-8161-f3113c1c6950" containerName="payload-processing"
Jul 24 00:50:51.907604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907598    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6806b69b-d571-408d-8e00-ff75deef3708" containerName="revoke-keys"
Jul 24 00:50:51.907604 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907605    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d133aa-f20f-4b03-8b09-7ebfe84dd8de" containerName="payload-pre-processing"
Jul 24 00:50:51.908196 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.907614    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="37deace1-899a-4797-ad10-6701adf07adb" containerName="maas-api"
Jul 24 00:50:51.910501 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.910477    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:51.913915 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.913882    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6a8188-serving-cert\""
Jul 24 00:50:51.938314 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.938276    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"]
Jul 24 00:50:51.977044 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.977003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbc59b8-804e-4f93-9f6c-4e37285be90e-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:51.977044 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.977052    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjls\" (UniqueName: \"kubernetes.io/projected/3dbc59b8-804e-4f93-9f6c-4e37285be90e-kube-api-access-kbjls\") pod \"maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:51.977255 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:51.977072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc59b8-804e-4f93-9f6c-4e37285be90e-maas-api-tls\") pod \"maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:52.077921 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.077878    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjls\" (UniqueName: \"kubernetes.io/projected/3dbc59b8-804e-4f93-9f6c-4e37285be90e-kube-api-access-kbjls\") pod \"maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:52.077921 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.077927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc59b8-804e-4f93-9f6c-4e37285be90e-maas-api-tls\") pod \"maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:52.078199 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:50:52.078107    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-serving-cert: secret "maas-api-e2e-api-a-6a8188-serving-cert" not found
Jul 24 00:50:52.078199 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:50:52.078191    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbc59b8-804e-4f93-9f6c-4e37285be90e-maas-api-tls podName:3dbc59b8-804e-4f93-9f6c-4e37285be90e nodeName:}" failed. No retries permitted until 2026-07-24 00:50:52.578166525 +0000 UTC m=+2233.145982658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3dbc59b8-804e-4f93-9f6c-4e37285be90e-maas-api-tls") pod "maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm" (UID: "3dbc59b8-804e-4f93-9f6c-4e37285be90e") : secret "maas-api-e2e-api-a-6a8188-serving-cert" not found
Jul 24 00:50:52.078334 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.078056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbc59b8-804e-4f93-9f6c-4e37285be90e-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:52.078867 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.078850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbc59b8-804e-4f93-9f6c-4e37285be90e-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:52.089417 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.089387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjls\" (UniqueName: \"kubernetes.io/projected/3dbc59b8-804e-4f93-9f6c-4e37285be90e-kube-api-access-kbjls\") pod \"maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:52.583058 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.583022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc59b8-804e-4f93-9f6c-4e37285be90e-maas-api-tls\") pod \"maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:52.585491 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.585464    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc59b8-804e-4f93-9f6c-4e37285be90e-maas-api-tls\") pod \"maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:52.657994 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.657949    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"]
Jul 24 00:50:52.661546 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.661522    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:50:52.663809 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.663790    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6a8188\""
Jul 24 00:50:52.672180 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.672153    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"]
Jul 24 00:50:52.700704 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.700670    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"]
Jul 24 00:50:52.703865 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.703849    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:50:52.706405 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.706386    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6a8188-dockercfg-4q9vg\""
Jul 24 00:50:52.714339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.714317    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"]
Jul 24 00:50:52.784824 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.784774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lxm\" (UniqueName: \"kubernetes.io/projected/6df971ee-0161-4c5c-a780-449f420626cf-kube-api-access-q5lxm\") pod \"payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r\" (UID: \"6df971ee-0161-4c5c-a780-449f420626cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:50:52.785034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.784930    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df971ee-0161-4c5c-a780-449f420626cf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r\" (UID: \"6df971ee-0161-4c5c-a780-449f420626cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:50:52.785034 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.785016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71db3479-de68-422f-b10d-eb3a0057f670-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl\" (UID: \"71db3479-de68-422f-b10d-eb3a0057f670\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:50:52.785118 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.785050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchxp\" (UniqueName: \"kubernetes.io/projected/71db3479-de68-422f-b10d-eb3a0057f670-kube-api-access-fchxp\") pod \"payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl\" (UID: \"71db3479-de68-422f-b10d-eb3a0057f670\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:50:52.821701 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.821656    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:52.886487 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.886433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71db3479-de68-422f-b10d-eb3a0057f670-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl\" (UID: \"71db3479-de68-422f-b10d-eb3a0057f670\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:50:52.886660 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.886501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchxp\" (UniqueName: \"kubernetes.io/projected/71db3479-de68-422f-b10d-eb3a0057f670-kube-api-access-fchxp\") pod \"payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl\" (UID: \"71db3479-de68-422f-b10d-eb3a0057f670\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:50:52.886660 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.886582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lxm\" (UniqueName: \"kubernetes.io/projected/6df971ee-0161-4c5c-a780-449f420626cf-kube-api-access-q5lxm\") pod \"payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r\" (UID: \"6df971ee-0161-4c5c-a780-449f420626cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:50:52.886778 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.886673    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df971ee-0161-4c5c-a780-449f420626cf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r\" (UID: \"6df971ee-0161-4c5c-a780-449f420626cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:50:52.887643 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.887608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df971ee-0161-4c5c-a780-449f420626cf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r\" (UID: \"6df971ee-0161-4c5c-a780-449f420626cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:50:52.888942 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.888903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71db3479-de68-422f-b10d-eb3a0057f670-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl\" (UID: \"71db3479-de68-422f-b10d-eb3a0057f670\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:50:52.896255 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.896223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchxp\" (UniqueName: \"kubernetes.io/projected/71db3479-de68-422f-b10d-eb3a0057f670-kube-api-access-fchxp\") pod \"payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl\" (UID: \"71db3479-de68-422f-b10d-eb3a0057f670\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:50:52.896655 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.896634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lxm\" (UniqueName: \"kubernetes.io/projected/6df971ee-0161-4c5c-a780-449f420626cf-kube-api-access-q5lxm\") pod \"payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r\" (UID: \"6df971ee-0161-4c5c-a780-449f420626cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:50:52.966409 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.966380    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"]
Jul 24 00:50:52.968690 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:50:52.968661    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbc59b8_804e_4f93_9f6c_4e37285be90e.slice/crio-20122396fb5f13234b66317287099079e7c9768e8deac0e32c10e3d717060b6f WatchSource:0}: Error finding container 20122396fb5f13234b66317287099079e7c9768e8deac0e32c10e3d717060b6f: Status 404 returned error can't find the container with id 20122396fb5f13234b66317287099079e7c9768e8deac0e32c10e3d717060b6f
Jul 24 00:50:52.969899 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.969883    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:50:52.972586 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:52.972563    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:50:53.014912 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:53.014856    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:50:53.112461 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:53.112420    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm" event={"ID":"3dbc59b8-804e-4f93-9f6c-4e37285be90e","Type":"ContainerStarted","Data":"20122396fb5f13234b66317287099079e7c9768e8deac0e32c10e3d717060b6f"}
Jul 24 00:50:53.119540 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:53.118623    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"]
Jul 24 00:50:53.127716 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:50:53.127617    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71db3479_de68_422f_b10d_eb3a0057f670.slice/crio-33caceaaf5788152992d404a53e6da7b23f5b2be9e7c397f7d09445e2f357125 WatchSource:0}: Error finding container 33caceaaf5788152992d404a53e6da7b23f5b2be9e7c397f7d09445e2f357125: Status 404 returned error can't find the container with id 33caceaaf5788152992d404a53e6da7b23f5b2be9e7c397f7d09445e2f357125
Jul 24 00:50:53.182258 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:53.182231    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"]
Jul 24 00:50:53.188987 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:50:53.188952    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df971ee_0161_4c5c_a780_449f420626cf.slice/crio-1c3cb571ff8ef99133d88747b95ae0de5fa3ebbb924b0ee7dee86852378cfd3d WatchSource:0}: Error finding container 1c3cb571ff8ef99133d88747b95ae0de5fa3ebbb924b0ee7dee86852378cfd3d: Status 404 returned error can't find the container with id 1c3cb571ff8ef99133d88747b95ae0de5fa3ebbb924b0ee7dee86852378cfd3d
Jul 24 00:50:54.126359 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:54.126259    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r" event={"ID":"6df971ee-0161-4c5c-a780-449f420626cf","Type":"ContainerStarted","Data":"1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a"}
Jul 24 00:50:54.126359 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:54.126355    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r" event={"ID":"6df971ee-0161-4c5c-a780-449f420626cf","Type":"ContainerStarted","Data":"1c3cb571ff8ef99133d88747b95ae0de5fa3ebbb924b0ee7dee86852378cfd3d"}
Jul 24 00:50:54.126902 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:54.126393    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:50:54.128326 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:54.128280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm" event={"ID":"3dbc59b8-804e-4f93-9f6c-4e37285be90e","Type":"ContainerStarted","Data":"20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21"}
Jul 24 00:50:54.128484 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:54.128448    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:50:54.130143 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:54.130112    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl" event={"ID":"71db3479-de68-422f-b10d-eb3a0057f670","Type":"ContainerStarted","Data":"94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c"}
Jul 24 00:50:54.130269 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:54.130164    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl" event={"ID":"71db3479-de68-422f-b10d-eb3a0057f670","Type":"ContainerStarted","Data":"33caceaaf5788152992d404a53e6da7b23f5b2be9e7c397f7d09445e2f357125"}
Jul 24 00:50:54.130269 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:54.130193    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:50:54.144891 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:54.144814    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r" podStartSLOduration=2.144793193 podStartE2EDuration="2.144793193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:50:54.14290592 +0000 UTC m=+2234.710722094" watchObservedRunningTime="2026-07-24 00:50:54.144793193 +0000 UTC m=+2234.712609343"
Jul 24 00:50:54.159332 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:54.159272    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm" podStartSLOduration=2.422621251 podStartE2EDuration="3.159255365s" totalImagesPullingTime="736.634114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:50:54.158106896 +0000 UTC m=+2234.725923049" watchObservedRunningTime="2026-07-24 00:50:54.159255365 +0000 UTC m=+2234.727071515"
Jul 24 00:50:54.175611 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:50:54.175557    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl" podStartSLOduration=2.175539631 podStartE2EDuration="2.175539631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:50:54.174245142 +0000 UTC m=+2234.742061293" watchObservedRunningTime="2026-07-24 00:50:54.175539631 +0000 UTC m=+2234.743355825"
Jul 24 00:51:00.000932 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.000893    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"]
Jul 24 00:51:00.005103 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.005075    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.008470 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.008444    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a43e61-serving-cert\""
Jul 24 00:51:00.030487 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.030453    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"]
Jul 24 00:51:00.058863 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.058811    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c66c6c-08e7-4359-9b6d-6a9949a68e74-maas-api-tls\") pod \"maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.058863 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.058873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ncf\" (UniqueName: \"kubernetes.io/projected/43c66c6c-08e7-4359-9b6d-6a9949a68e74-kube-api-access-v5ncf\") pod \"maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.059216 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.058991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c66c6c-08e7-4359-9b6d-6a9949a68e74-openshift-service-ca\") pod \"maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.140824 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.140794    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:51:00.159488 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.159453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c66c6c-08e7-4359-9b6d-6a9949a68e74-openshift-service-ca\") pod \"maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.159677 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.159539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c66c6c-08e7-4359-9b6d-6a9949a68e74-maas-api-tls\") pod \"maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.159677 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.159567    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ncf\" (UniqueName: \"kubernetes.io/projected/43c66c6c-08e7-4359-9b6d-6a9949a68e74-kube-api-access-v5ncf\") pod \"maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.159774 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:51:00.159728    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-serving-cert: secret "maas-api-e2e-api-b-a43e61-serving-cert" not found
Jul 24 00:51:00.159818 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:51:00.159803    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43c66c6c-08e7-4359-9b6d-6a9949a68e74-maas-api-tls podName:43c66c6c-08e7-4359-9b6d-6a9949a68e74 nodeName:}" failed. No retries permitted until 2026-07-24 00:51:00.659780324 +0000 UTC m=+2241.227596451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43c66c6c-08e7-4359-9b6d-6a9949a68e74-maas-api-tls") pod "maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j" (UID: "43c66c6c-08e7-4359-9b6d-6a9949a68e74") : secret "maas-api-e2e-api-b-a43e61-serving-cert" not found
Jul 24 00:51:00.160274 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.160256    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c66c6c-08e7-4359-9b6d-6a9949a68e74-openshift-service-ca\") pod \"maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.170760 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.170721    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ncf\" (UniqueName: \"kubernetes.io/projected/43c66c6c-08e7-4359-9b6d-6a9949a68e74-kube-api-access-v5ncf\") pod \"maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.665181 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.665133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c66c6c-08e7-4359-9b6d-6a9949a68e74-maas-api-tls\") pod \"maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.667626 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.667592    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c66c6c-08e7-4359-9b6d-6a9949a68e74-maas-api-tls\") pod \"maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.857185 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.857148    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"]
Jul 24 00:51:00.860648 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.860631    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:00.863072 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.863048    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a43e61-dockercfg-dgxdc\""
Jul 24 00:51:00.863219 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.863097    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a43e61\""
Jul 24 00:51:00.875151 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.875123    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"]
Jul 24 00:51:00.915637 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.915544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:00.920893 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.920865    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"]
Jul 24 00:51:00.925252 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.925227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:00.932847 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.932586    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"]
Jul 24 00:51:00.967601 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.967570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a43e61-567485c5b6-844sx\" (UID: \"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:00.967771 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:00.967637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzp5s\" (UniqueName: \"kubernetes.io/projected/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-kube-api-access-vzp5s\") pod \"payload-processing-e2e-api-b-a43e61-567485c5b6-844sx\" (UID: \"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:01.068497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.068456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a43e61-567485c5b6-844sx\" (UID: \"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:01.068960 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.068551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzp5s\" (UniqueName: \"kubernetes.io/projected/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-kube-api-access-vzp5s\") pod \"payload-processing-e2e-api-b-a43e61-567485c5b6-844sx\" (UID: \"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:01.068960 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.068649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63deaf01-b0c3-4cb7-baf5-d943345a1834-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46\" (UID: \"63deaf01-b0c3-4cb7-baf5-d943345a1834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:01.068960 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.068677    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c252\" (UniqueName: \"kubernetes.io/projected/63deaf01-b0c3-4cb7-baf5-d943345a1834-kube-api-access-4c252\") pod \"payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46\" (UID: \"63deaf01-b0c3-4cb7-baf5-d943345a1834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:01.069496 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.069469    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a43e61-567485c5b6-844sx\" (UID: \"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:01.073530 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.073505    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"]
Jul 24 00:51:01.075497 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:51:01.075466    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c66c6c_08e7_4359_9b6d_6a9949a68e74.slice/crio-7f7bfc2b57fb0f1607135858d85ee90d1a2e5601ac63241826b12ae90c5ba107 WatchSource:0}: Error finding container 7f7bfc2b57fb0f1607135858d85ee90d1a2e5601ac63241826b12ae90c5ba107: Status 404 returned error can't find the container with id 7f7bfc2b57fb0f1607135858d85ee90d1a2e5601ac63241826b12ae90c5ba107
Jul 24 00:51:01.078151 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.078127    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzp5s\" (UniqueName: \"kubernetes.io/projected/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-kube-api-access-vzp5s\") pod \"payload-processing-e2e-api-b-a43e61-567485c5b6-844sx\" (UID: \"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4\") " pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:01.167143 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.167060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j" event={"ID":"43c66c6c-08e7-4359-9b6d-6a9949a68e74","Type":"ContainerStarted","Data":"7f7bfc2b57fb0f1607135858d85ee90d1a2e5601ac63241826b12ae90c5ba107"}
Jul 24 00:51:01.169539 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.169508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63deaf01-b0c3-4cb7-baf5-d943345a1834-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46\" (UID: \"63deaf01-b0c3-4cb7-baf5-d943345a1834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:01.169681 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.169545    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c252\" (UniqueName: \"kubernetes.io/projected/63deaf01-b0c3-4cb7-baf5-d943345a1834-kube-api-access-4c252\") pod \"payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46\" (UID: \"63deaf01-b0c3-4cb7-baf5-d943345a1834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:01.170402 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.170375    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63deaf01-b0c3-4cb7-baf5-d943345a1834-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46\" (UID: \"63deaf01-b0c3-4cb7-baf5-d943345a1834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:01.171428 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.171407    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:01.180066 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.179987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c252\" (UniqueName: \"kubernetes.io/projected/63deaf01-b0c3-4cb7-baf5-d943345a1834-kube-api-access-4c252\") pod \"payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46\" (UID: \"63deaf01-b0c3-4cb7-baf5-d943345a1834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:01.253640 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.253603    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:01.336417 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.336232    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"]
Jul 24 00:51:01.419563 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:01.419469    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"]
Jul 24 00:51:01.423395 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:51:01.423362    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63deaf01_b0c3_4cb7_baf5_d943345a1834.slice/crio-db4dd65c2ac8ab1ee144d22122468282177dedb491f56988189cc7a748eecdaa WatchSource:0}: Error finding container db4dd65c2ac8ab1ee144d22122468282177dedb491f56988189cc7a748eecdaa: Status 404 returned error can't find the container with id db4dd65c2ac8ab1ee144d22122468282177dedb491f56988189cc7a748eecdaa
Jul 24 00:51:02.174373 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:02.174288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx" event={"ID":"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4","Type":"ContainerStarted","Data":"e1a5284bda2e13e1160e63d4c305e41e35870bda0505e696bec49d8c0e4ad31a"}
Jul 24 00:51:02.174373 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:02.174351    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx" event={"ID":"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4","Type":"ContainerStarted","Data":"fbc36b574025540852ee15118967a2e7d2cd670503f6702a83c0562a79b8dd6e"}
Jul 24 00:51:02.174879 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:02.174407    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:02.176229 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:02.176205    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j" event={"ID":"43c66c6c-08e7-4359-9b6d-6a9949a68e74","Type":"ContainerStarted","Data":"8f20da40330570bce46bbc9c7e4258d72ac5886bcdc77404addfdb31f90ed3e7"}
Jul 24 00:51:02.176373 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:02.176255    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:02.177993 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:02.177969    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46" event={"ID":"63deaf01-b0c3-4cb7-baf5-d943345a1834","Type":"ContainerStarted","Data":"19629e6494df3ad755327f23b03e5874757621696d5cf5c296de8d3c9d6d6da5"}
Jul 24 00:51:02.177993 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:02.178003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46" event={"ID":"63deaf01-b0c3-4cb7-baf5-d943345a1834","Type":"ContainerStarted","Data":"db4dd65c2ac8ab1ee144d22122468282177dedb491f56988189cc7a748eecdaa"}
Jul 24 00:51:02.178191 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:02.178134    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:02.193190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:02.193142    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx" podStartSLOduration=2.193126909 podStartE2EDuration="2.193126909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:51:02.190481927 +0000 UTC m=+2242.758298077" watchObservedRunningTime="2026-07-24 00:51:02.193126909 +0000 UTC m=+2242.760943059"
Jul 24 00:51:02.207689 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:02.207636    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j" podStartSLOduration=2.726132045 podStartE2EDuration="3.207617394s" totalImagesPullingTime="481.485349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:51:02.207090887 +0000 UTC m=+2242.774907037" watchObservedRunningTime="2026-07-24 00:51:02.207617394 +0000 UTC m=+2242.775433545"
Jul 24 00:51:02.224205 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:02.224158    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46" podStartSLOduration=2.224142907 podStartE2EDuration="2.224142907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:51:02.222099999 +0000 UTC m=+2242.789916151" watchObservedRunningTime="2026-07-24 00:51:02.224142907 +0000 UTC m=+2242.791959057"
Jul 24 00:51:05.138363 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:05.138319    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:51:05.138813 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:05.138490    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:51:08.188491 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:08.188460    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:13.184695 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:13.184660    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:13.187075 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:13.184853    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:28.344252 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.344202    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"]
Jul 24 00:51:28.344726 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.344506    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm" podUID="3dbc59b8-804e-4f93-9f6c-4e37285be90e" containerName="maas-api" containerID="cri-o://20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21" gracePeriod=30
Jul 24 00:51:28.398305 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.398275    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"]
Jul 24 00:51:28.398541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.398516    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r" podUID="6df971ee-0161-4c5c-a780-449f420626cf" containerName="payload-processing" containerID="cri-o://1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a" gracePeriod=30
Jul 24 00:51:28.406437 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.406286    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"]
Jul 24 00:51:28.406575 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.406530    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl" podUID="71db3479-de68-422f-b10d-eb3a0057f670" containerName="payload-pre-processing" containerID="cri-o://94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c" gracePeriod=30
Jul 24 00:51:28.674578 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.674550    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:51:28.692017 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.691796    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:51:28.696297 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.696272    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:51:28.819325 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.819283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbjls\" (UniqueName: \"kubernetes.io/projected/3dbc59b8-804e-4f93-9f6c-4e37285be90e-kube-api-access-kbjls\") pod \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") "
Jul 24 00:51:28.819528 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.819370    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df971ee-0161-4c5c-a780-449f420626cf-plugins-config-volume\") pod \"6df971ee-0161-4c5c-a780-449f420626cf\" (UID: \"6df971ee-0161-4c5c-a780-449f420626cf\") "
Jul 24 00:51:28.819528 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.819409    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71db3479-de68-422f-b10d-eb3a0057f670-plugins-config-volume\") pod \"71db3479-de68-422f-b10d-eb3a0057f670\" (UID: \"71db3479-de68-422f-b10d-eb3a0057f670\") "
Jul 24 00:51:28.819528 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.819438    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5lxm\" (UniqueName: \"kubernetes.io/projected/6df971ee-0161-4c5c-a780-449f420626cf-kube-api-access-q5lxm\") pod \"6df971ee-0161-4c5c-a780-449f420626cf\" (UID: \"6df971ee-0161-4c5c-a780-449f420626cf\") "
Jul 24 00:51:28.819528 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.819461    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fchxp\" (UniqueName: \"kubernetes.io/projected/71db3479-de68-422f-b10d-eb3a0057f670-kube-api-access-fchxp\") pod \"71db3479-de68-422f-b10d-eb3a0057f670\" (UID: \"71db3479-de68-422f-b10d-eb3a0057f670\") "
Jul 24 00:51:28.819764 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.819618    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbc59b8-804e-4f93-9f6c-4e37285be90e-openshift-service-ca\") pod \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") "
Jul 24 00:51:28.820079 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.820060    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc59b8-804e-4f93-9f6c-4e37285be90e-maas-api-tls\") pod \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\" (UID: \"3dbc59b8-804e-4f93-9f6c-4e37285be90e\") "
Jul 24 00:51:28.820192 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.819870    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71db3479-de68-422f-b10d-eb3a0057f670-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71db3479-de68-422f-b10d-eb3a0057f670" (UID: "71db3479-de68-422f-b10d-eb3a0057f670"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:51:28.820192 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.819952    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df971ee-0161-4c5c-a780-449f420626cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6df971ee-0161-4c5c-a780-449f420626cf" (UID: "6df971ee-0161-4c5c-a780-449f420626cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:51:28.820192 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.819991    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbc59b8-804e-4f93-9f6c-4e37285be90e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3dbc59b8-804e-4f93-9f6c-4e37285be90e" (UID: "3dbc59b8-804e-4f93-9f6c-4e37285be90e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:51:28.820361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.820301    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df971ee-0161-4c5c-a780-449f420626cf-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:28.820361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.820316    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71db3479-de68-422f-b10d-eb3a0057f670-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:28.820361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.820325    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbc59b8-804e-4f93-9f6c-4e37285be90e-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:28.821805 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.821755    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbc59b8-804e-4f93-9f6c-4e37285be90e-kube-api-access-kbjls" (OuterVolumeSpecName: "kube-api-access-kbjls") pod "3dbc59b8-804e-4f93-9f6c-4e37285be90e" (UID: "3dbc59b8-804e-4f93-9f6c-4e37285be90e"). InnerVolumeSpecName "kube-api-access-kbjls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:51:28.821805 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.821765    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71db3479-de68-422f-b10d-eb3a0057f670-kube-api-access-fchxp" (OuterVolumeSpecName: "kube-api-access-fchxp") pod "71db3479-de68-422f-b10d-eb3a0057f670" (UID: "71db3479-de68-422f-b10d-eb3a0057f670"). InnerVolumeSpecName "kube-api-access-fchxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:51:28.822014 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.821992    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df971ee-0161-4c5c-a780-449f420626cf-kube-api-access-q5lxm" (OuterVolumeSpecName: "kube-api-access-q5lxm") pod "6df971ee-0161-4c5c-a780-449f420626cf" (UID: "6df971ee-0161-4c5c-a780-449f420626cf"). InnerVolumeSpecName "kube-api-access-q5lxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:51:28.822185 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.822165    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbc59b8-804e-4f93-9f6c-4e37285be90e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3dbc59b8-804e-4f93-9f6c-4e37285be90e" (UID: "3dbc59b8-804e-4f93-9f6c-4e37285be90e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:51:28.921249 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.921161    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5lxm\" (UniqueName: \"kubernetes.io/projected/6df971ee-0161-4c5c-a780-449f420626cf-kube-api-access-q5lxm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:28.921249 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.921192    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fchxp\" (UniqueName: \"kubernetes.io/projected/71db3479-de68-422f-b10d-eb3a0057f670-kube-api-access-fchxp\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:28.921249 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.921202    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc59b8-804e-4f93-9f6c-4e37285be90e-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:28.921249 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:28.921211    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbjls\" (UniqueName: \"kubernetes.io/projected/3dbc59b8-804e-4f93-9f6c-4e37285be90e-kube-api-access-kbjls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:29.309310 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.309271    2560 generic.go:356] "Generic (PLEG): container finished" podID="6df971ee-0161-4c5c-a780-449f420626cf" containerID="1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a" exitCode=0
Jul 24 00:51:29.309502 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.309331    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"
Jul 24 00:51:29.309502 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.309351    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r" event={"ID":"6df971ee-0161-4c5c-a780-449f420626cf","Type":"ContainerDied","Data":"1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a"}
Jul 24 00:51:29.309502 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.309413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r" event={"ID":"6df971ee-0161-4c5c-a780-449f420626cf","Type":"ContainerDied","Data":"1c3cb571ff8ef99133d88747b95ae0de5fa3ebbb924b0ee7dee86852378cfd3d"}
Jul 24 00:51:29.309502 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.309443    2560 scope.go:117] "RemoveContainer" containerID="1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a"
Jul 24 00:51:29.310766 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.310743    2560 generic.go:356] "Generic (PLEG): container finished" podID="3dbc59b8-804e-4f93-9f6c-4e37285be90e" containerID="20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21" exitCode=0
Jul 24 00:51:29.310877 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.310858    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"
Jul 24 00:51:29.310939 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.310871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm" event={"ID":"3dbc59b8-804e-4f93-9f6c-4e37285be90e","Type":"ContainerDied","Data":"20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21"}
Jul 24 00:51:29.310939 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.310928    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm" event={"ID":"3dbc59b8-804e-4f93-9f6c-4e37285be90e","Type":"ContainerDied","Data":"20122396fb5f13234b66317287099079e7c9768e8deac0e32c10e3d717060b6f"}
Jul 24 00:51:29.312240 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.312212    2560 generic.go:356] "Generic (PLEG): container finished" podID="71db3479-de68-422f-b10d-eb3a0057f670" containerID="94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c" exitCode=0
Jul 24 00:51:29.312351 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.312239    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl" event={"ID":"71db3479-de68-422f-b10d-eb3a0057f670","Type":"ContainerDied","Data":"94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c"}
Jul 24 00:51:29.312351 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.312276    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"
Jul 24 00:51:29.312351 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.312285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl" event={"ID":"71db3479-de68-422f-b10d-eb3a0057f670","Type":"ContainerDied","Data":"33caceaaf5788152992d404a53e6da7b23f5b2be9e7c397f7d09445e2f357125"}
Jul 24 00:51:29.320071 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.320054    2560 scope.go:117] "RemoveContainer" containerID="1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a"
Jul 24 00:51:29.320368 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:51:29.320342    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a\": container with ID starting with 1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a not found: ID does not exist" containerID="1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a"
Jul 24 00:51:29.320473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.320379    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a"} err="failed to get container status \"1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a\": rpc error: code = NotFound desc = could not find container \"1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a\": container with ID starting with 1dcbce1470d7efe6e76ad5cfb053db197f6111dd17cdff4aa6baba1920622e9a not found: ID does not exist"
Jul 24 00:51:29.320473 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.320404    2560 scope.go:117] "RemoveContainer" containerID="20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21"
Jul 24 00:51:29.329778 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.329760    2560 scope.go:117] "RemoveContainer" containerID="20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21"
Jul 24 00:51:29.330032 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:51:29.330016    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21\": container with ID starting with 20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21 not found: ID does not exist" containerID="20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21"
Jul 24 00:51:29.330077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.330039    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21"} err="failed to get container status \"20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21\": rpc error: code = NotFound desc = could not find container \"20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21\": container with ID starting with 20f8b2ace94ed088c854df7079241bd67d599841b8a96d17e892e065a420fc21 not found: ID does not exist"
Jul 24 00:51:29.330077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.330057    2560 scope.go:117] "RemoveContainer" containerID="94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c"
Jul 24 00:51:29.336658 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.336635    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"]
Jul 24 00:51:29.339325 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.339306    2560 scope.go:117] "RemoveContainer" containerID="94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c"
Jul 24 00:51:29.339643 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:51:29.339618    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c\": container with ID starting with 94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c not found: ID does not exist" containerID="94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c"
Jul 24 00:51:29.339691 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.339653    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c"} err="failed to get container status \"94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c\": rpc error: code = NotFound desc = could not find container \"94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c\": container with ID starting with 94078cd18385005da98b319767b663063bc5e49248e5b1e350931ba2c4ac564c not found: ID does not exist"
Jul 24 00:51:29.340674 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.340653    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8188-5cfb9c46cb-7mz9r"]
Jul 24 00:51:29.349894 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.349869    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"]
Jul 24 00:51:29.352627 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.352609    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8188-797bf4bd84-hj4dm"]
Jul 24 00:51:29.361419 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.361267    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"]
Jul 24 00:51:29.367533 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.367509    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8188-58b8cdccd8-5dbhl"]
Jul 24 00:51:29.968186 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.968154    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dbc59b8-804e-4f93-9f6c-4e37285be90e" path="/var/lib/kubelet/pods/3dbc59b8-804e-4f93-9f6c-4e37285be90e/volumes"
Jul 24 00:51:29.968497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.968484    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df971ee-0161-4c5c-a780-449f420626cf" path="/var/lib/kubelet/pods/6df971ee-0161-4c5c-a780-449f420626cf/volumes"
Jul 24 00:51:29.968789 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:29.968778    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71db3479-de68-422f-b10d-eb3a0057f670" path="/var/lib/kubelet/pods/71db3479-de68-422f-b10d-eb3a0057f670/volumes"
Jul 24 00:51:39.775891 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.775846    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"]
Jul 24 00:51:39.776497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.776216    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71db3479-de68-422f-b10d-eb3a0057f670" containerName="payload-pre-processing"
Jul 24 00:51:39.776497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.776228    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="71db3479-de68-422f-b10d-eb3a0057f670" containerName="payload-pre-processing"
Jul 24 00:51:39.776497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.776262    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df971ee-0161-4c5c-a780-449f420626cf" containerName="payload-processing"
Jul 24 00:51:39.776497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.776269    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df971ee-0161-4c5c-a780-449f420626cf" containerName="payload-processing"
Jul 24 00:51:39.776497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.776281    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dbc59b8-804e-4f93-9f6c-4e37285be90e" containerName="maas-api"
Jul 24 00:51:39.776497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.776288    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dbc59b8-804e-4f93-9f6c-4e37285be90e" containerName="maas-api"
Jul 24 00:51:39.776497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.776342    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df971ee-0161-4c5c-a780-449f420626cf" containerName="payload-processing"
Jul 24 00:51:39.776497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.776355    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dbc59b8-804e-4f93-9f6c-4e37285be90e" containerName="maas-api"
Jul 24 00:51:39.776497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.776361    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="71db3479-de68-422f-b10d-eb3a0057f670" containerName="payload-pre-processing"
Jul 24 00:51:39.781024 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.780996    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"
Jul 24 00:51:39.783166 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.783139    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\""
Jul 24 00:51:39.789260 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.789225    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"]
Jul 24 00:51:39.813804 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.813766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c0104-ef40-43c7-921b-db247b31566c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc\" (UID: \"2a8c0104-ef40-43c7-921b-db247b31566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"
Jul 24 00:51:39.915150 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.915110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c0104-ef40-43c7-921b-db247b31566c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc\" (UID: \"2a8c0104-ef40-43c7-921b-db247b31566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"
Jul 24 00:51:39.915805 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:39.915781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c0104-ef40-43c7-921b-db247b31566c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc\" (UID: \"2a8c0104-ef40-43c7-921b-db247b31566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"
Jul 24 00:51:40.105125 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:40.105051    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\""
Jul 24 00:51:40.113838 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:40.113803    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"
Jul 24 00:51:40.266347 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:40.266314    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"]
Jul 24 00:51:40.269589 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:51:40.269559    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8c0104_ef40_43c7_921b_db247b31566c.slice/crio-5be54c2e4fa0dd13cfaf2d8ad8435872c17f74efa9cb648192266fc64668e5f7 WatchSource:0}: Error finding container 5be54c2e4fa0dd13cfaf2d8ad8435872c17f74efa9cb648192266fc64668e5f7: Status 404 returned error can't find the container with id 5be54c2e4fa0dd13cfaf2d8ad8435872c17f74efa9cb648192266fc64668e5f7
Jul 24 00:51:40.369106 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:40.369027    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc" event={"ID":"2a8c0104-ef40-43c7-921b-db247b31566c","Type":"ContainerStarted","Data":"5be54c2e4fa0dd13cfaf2d8ad8435872c17f74efa9cb648192266fc64668e5f7"}
Jul 24 00:51:41.374533 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:41.374499    2560 generic.go:356] "Generic (PLEG): container finished" podID="2a8c0104-ef40-43c7-921b-db247b31566c" containerID="5b875feaf7c3b98012eef29288a183094ec1b07395b55ff04e2a4d7f631caf1f" exitCode=0
Jul 24 00:51:41.374946 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:41.374565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc" event={"ID":"2a8c0104-ef40-43c7-921b-db247b31566c","Type":"ContainerDied","Data":"5b875feaf7c3b98012eef29288a183094ec1b07395b55ff04e2a4d7f631caf1f"}
Jul 24 00:51:42.509211 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:42.509180    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"
Jul 24 00:51:42.538749 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:42.538717    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c0104-ef40-43c7-921b-db247b31566c-maas-api-service-ca\") pod \"2a8c0104-ef40-43c7-921b-db247b31566c\" (UID: \"2a8c0104-ef40-43c7-921b-db247b31566c\") "
Jul 24 00:51:42.539061 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:42.539038    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8c0104-ef40-43c7-921b-db247b31566c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a8c0104-ef40-43c7-921b-db247b31566c" (UID: "2a8c0104-ef40-43c7-921b-db247b31566c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:51:42.639730 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:42.639640    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c0104-ef40-43c7-921b-db247b31566c-maas-api-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:43.385425 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:43.385392    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"
Jul 24 00:51:43.385607 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:43.385436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc" event={"ID":"2a8c0104-ef40-43c7-921b-db247b31566c","Type":"ContainerDied","Data":"5be54c2e4fa0dd13cfaf2d8ad8435872c17f74efa9cb648192266fc64668e5f7"}
Jul 24 00:51:43.385607 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:43.385490    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5be54c2e4fa0dd13cfaf2d8ad8435872c17f74efa9cb648192266fc64668e5f7"
Jul 24 00:51:55.189712 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.189415    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"]
Jul 24 00:51:55.190259 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.190074    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j" podUID="43c66c6c-08e7-4359-9b6d-6a9949a68e74" containerName="maas-api" containerID="cri-o://8f20da40330570bce46bbc9c7e4258d72ac5886bcdc77404addfdb31f90ed3e7" gracePeriod=30
Jul 24 00:51:55.244729 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.244693    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"]
Jul 24 00:51:55.245028 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.244996    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx" podUID="f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4" containerName="payload-processing" containerID="cri-o://e1a5284bda2e13e1160e63d4c305e41e35870bda0505e696bec49d8c0e4ad31a" gracePeriod=30
Jul 24 00:51:55.250544 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.250505    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"]
Jul 24 00:51:55.250746 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.250717    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46" podUID="63deaf01-b0c3-4cb7-baf5-d943345a1834" containerName="payload-pre-processing" containerID="cri-o://19629e6494df3ad755327f23b03e5874757621696d5cf5c296de8d3c9d6d6da5" gracePeriod=30
Jul 24 00:51:55.444307 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.444226    2560 generic.go:356] "Generic (PLEG): container finished" podID="63deaf01-b0c3-4cb7-baf5-d943345a1834" containerID="19629e6494df3ad755327f23b03e5874757621696d5cf5c296de8d3c9d6d6da5" exitCode=0
Jul 24 00:51:55.444462 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.444298    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46" event={"ID":"63deaf01-b0c3-4cb7-baf5-d943345a1834","Type":"ContainerDied","Data":"19629e6494df3ad755327f23b03e5874757621696d5cf5c296de8d3c9d6d6da5"}
Jul 24 00:51:55.445880 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.445857    2560 generic.go:356] "Generic (PLEG): container finished" podID="f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4" containerID="e1a5284bda2e13e1160e63d4c305e41e35870bda0505e696bec49d8c0e4ad31a" exitCode=0
Jul 24 00:51:55.446009 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.445858    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx" event={"ID":"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4","Type":"ContainerDied","Data":"e1a5284bda2e13e1160e63d4c305e41e35870bda0505e696bec49d8c0e4ad31a"}
Jul 24 00:51:55.447430 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.447375    2560 generic.go:356] "Generic (PLEG): container finished" podID="43c66c6c-08e7-4359-9b6d-6a9949a68e74" containerID="8f20da40330570bce46bbc9c7e4258d72ac5886bcdc77404addfdb31f90ed3e7" exitCode=0
Jul 24 00:51:55.447554 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.447447    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j" event={"ID":"43c66c6c-08e7-4359-9b6d-6a9949a68e74","Type":"ContainerDied","Data":"8f20da40330570bce46bbc9c7e4258d72ac5886bcdc77404addfdb31f90ed3e7"}
Jul 24 00:51:55.525608 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.525582    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:55.545665 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.545636    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:55.547644 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.547621    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:55.651603 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.651572    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c66c6c-08e7-4359-9b6d-6a9949a68e74-maas-api-tls\") pod \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") "
Jul 24 00:51:55.651767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.651613    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5ncf\" (UniqueName: \"kubernetes.io/projected/43c66c6c-08e7-4359-9b6d-6a9949a68e74-kube-api-access-v5ncf\") pod \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") "
Jul 24 00:51:55.651767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.651651    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzp5s\" (UniqueName: \"kubernetes.io/projected/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-kube-api-access-vzp5s\") pod \"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4\" (UID: \"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4\") "
Jul 24 00:51:55.651767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.651669    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c252\" (UniqueName: \"kubernetes.io/projected/63deaf01-b0c3-4cb7-baf5-d943345a1834-kube-api-access-4c252\") pod \"63deaf01-b0c3-4cb7-baf5-d943345a1834\" (UID: \"63deaf01-b0c3-4cb7-baf5-d943345a1834\") "
Jul 24 00:51:55.651767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.651702    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63deaf01-b0c3-4cb7-baf5-d943345a1834-plugins-config-volume\") pod \"63deaf01-b0c3-4cb7-baf5-d943345a1834\" (UID: \"63deaf01-b0c3-4cb7-baf5-d943345a1834\") "
Jul 24 00:51:55.651767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.651724    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-plugins-config-volume\") pod \"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4\" (UID: \"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4\") "
Jul 24 00:51:55.652125 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.651822    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c66c6c-08e7-4359-9b6d-6a9949a68e74-openshift-service-ca\") pod \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\" (UID: \"43c66c6c-08e7-4359-9b6d-6a9949a68e74\") "
Jul 24 00:51:55.652194 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.652129    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4" (UID: "f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:51:55.652194 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.652148    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c66c6c-08e7-4359-9b6d-6a9949a68e74-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43c66c6c-08e7-4359-9b6d-6a9949a68e74" (UID: "43c66c6c-08e7-4359-9b6d-6a9949a68e74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:51:55.652286 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.652222    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63deaf01-b0c3-4cb7-baf5-d943345a1834-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63deaf01-b0c3-4cb7-baf5-d943345a1834" (UID: "63deaf01-b0c3-4cb7-baf5-d943345a1834"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 00:51:55.653870 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.653844    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c66c6c-08e7-4359-9b6d-6a9949a68e74-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43c66c6c-08e7-4359-9b6d-6a9949a68e74" (UID: "43c66c6c-08e7-4359-9b6d-6a9949a68e74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:51:55.654044 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.654024    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63deaf01-b0c3-4cb7-baf5-d943345a1834-kube-api-access-4c252" (OuterVolumeSpecName: "kube-api-access-4c252") pod "63deaf01-b0c3-4cb7-baf5-d943345a1834" (UID: "63deaf01-b0c3-4cb7-baf5-d943345a1834"). InnerVolumeSpecName "kube-api-access-4c252". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:51:55.654112 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.654063    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-kube-api-access-vzp5s" (OuterVolumeSpecName: "kube-api-access-vzp5s") pod "f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4" (UID: "f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4"). InnerVolumeSpecName "kube-api-access-vzp5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:51:55.654153 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.654139    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c66c6c-08e7-4359-9b6d-6a9949a68e74-kube-api-access-v5ncf" (OuterVolumeSpecName: "kube-api-access-v5ncf") pod "43c66c6c-08e7-4359-9b6d-6a9949a68e74" (UID: "43c66c6c-08e7-4359-9b6d-6a9949a68e74"). InnerVolumeSpecName "kube-api-access-v5ncf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:51:55.752660 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.752628    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c66c6c-08e7-4359-9b6d-6a9949a68e74-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:55.752660 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.752655    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c66c6c-08e7-4359-9b6d-6a9949a68e74-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:55.752660 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.752666    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5ncf\" (UniqueName: \"kubernetes.io/projected/43c66c6c-08e7-4359-9b6d-6a9949a68e74-kube-api-access-v5ncf\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:55.752917 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.752676    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzp5s\" (UniqueName: \"kubernetes.io/projected/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-kube-api-access-vzp5s\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:55.752917 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.752686    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c252\" (UniqueName: \"kubernetes.io/projected/63deaf01-b0c3-4cb7-baf5-d943345a1834-kube-api-access-4c252\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:55.752917 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.752694    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63deaf01-b0c3-4cb7-baf5-d943345a1834-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:55.752917 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:55.752703    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:51:56.453138 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.453047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx" event={"ID":"f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4","Type":"ContainerDied","Data":"fbc36b574025540852ee15118967a2e7d2cd670503f6702a83c0562a79b8dd6e"}
Jul 24 00:51:56.453138 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.453120    2560 scope.go:117] "RemoveContainer" containerID="e1a5284bda2e13e1160e63d4c305e41e35870bda0505e696bec49d8c0e4ad31a"
Jul 24 00:51:56.453608 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.453060    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"
Jul 24 00:51:56.454510 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.454476    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"
Jul 24 00:51:56.454586 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.454507    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j" event={"ID":"43c66c6c-08e7-4359-9b6d-6a9949a68e74","Type":"ContainerDied","Data":"7f7bfc2b57fb0f1607135858d85ee90d1a2e5601ac63241826b12ae90c5ba107"}
Jul 24 00:51:56.456077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.456049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46" event={"ID":"63deaf01-b0c3-4cb7-baf5-d943345a1834","Type":"ContainerDied","Data":"db4dd65c2ac8ab1ee144d22122468282177dedb491f56988189cc7a748eecdaa"}
Jul 24 00:51:56.456190 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.456124    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"
Jul 24 00:51:56.462663 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.462647    2560 scope.go:117] "RemoveContainer" containerID="8f20da40330570bce46bbc9c7e4258d72ac5886bcdc77404addfdb31f90ed3e7"
Jul 24 00:51:56.470231 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.470215    2560 scope.go:117] "RemoveContainer" containerID="19629e6494df3ad755327f23b03e5874757621696d5cf5c296de8d3c9d6d6da5"
Jul 24 00:51:56.474542 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.474514    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"]
Jul 24 00:51:56.479079 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.479060    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a43e61-6974b6f8dd-s8h46"]
Jul 24 00:51:56.487725 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.487702    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"]
Jul 24 00:51:56.490414 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.490393    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43e61-6fb4ddcf7c-2vx9j"]
Jul 24 00:51:56.499424 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.499400    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"]
Jul 24 00:51:56.503110 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:56.503086    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a43e61-567485c5b6-844sx"]
Jul 24 00:51:57.967364 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:57.967329    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c66c6c-08e7-4359-9b6d-6a9949a68e74" path="/var/lib/kubelet/pods/43c66c6c-08e7-4359-9b6d-6a9949a68e74/volumes"
Jul 24 00:51:57.967719 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:57.967677    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63deaf01-b0c3-4cb7-baf5-d943345a1834" path="/var/lib/kubelet/pods/63deaf01-b0c3-4cb7-baf5-d943345a1834/volumes"
Jul 24 00:51:57.968032 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:57.968019    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4" path="/var/lib/kubelet/pods/f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4/volumes"
Jul 24 00:51:59.856059 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:59.856020    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"]
Jul 24 00:51:59.857464 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:59.857441    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a43e61-n9ccc"]
Jul 24 00:51:59.967440 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:51:59.967405    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a8c0104-ef40-43c7-921b-db247b31566c" path="/var/lib/kubelet/pods/2a8c0104-ef40-43c7-921b-db247b31566c/volumes"
Jul 24 00:52:10.401465 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401420    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"]
Jul 24 00:52:10.401898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401811    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63deaf01-b0c3-4cb7-baf5-d943345a1834" containerName="payload-pre-processing"
Jul 24 00:52:10.401898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401825    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="63deaf01-b0c3-4cb7-baf5-d943345a1834" containerName="payload-pre-processing"
Jul 24 00:52:10.401898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401860    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c66c6c-08e7-4359-9b6d-6a9949a68e74" containerName="maas-api"
Jul 24 00:52:10.401898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401865    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c66c6c-08e7-4359-9b6d-6a9949a68e74" containerName="maas-api"
Jul 24 00:52:10.401898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401872    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8c0104-ef40-43c7-921b-db247b31566c" containerName="revoke-keys"
Jul 24 00:52:10.401898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401877    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8c0104-ef40-43c7-921b-db247b31566c" containerName="revoke-keys"
Jul 24 00:52:10.401898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401886    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4" containerName="payload-processing"
Jul 24 00:52:10.401898 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401891    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4" containerName="payload-processing"
Jul 24 00:52:10.402168 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401944    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2dbdcbe-c3f2-4c68-a50c-1fe0c173efb4" containerName="payload-processing"
Jul 24 00:52:10.402168 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401951    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a8c0104-ef40-43c7-921b-db247b31566c" containerName="revoke-keys"
Jul 24 00:52:10.402168 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401958    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="63deaf01-b0c3-4cb7-baf5-d943345a1834" containerName="payload-pre-processing"
Jul 24 00:52:10.402168 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.401968    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c66c6c-08e7-4359-9b6d-6a9949a68e74" containerName="maas-api"
Jul 24 00:52:10.406448 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.406429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:10.408959 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.408935    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-38da25-serving-cert\""
Jul 24 00:52:10.414687 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.414658    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"]
Jul 24 00:52:10.480521 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.480481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76848b08-90c1-4723-85c5-3a876bb02b0e-maas-api-tls\") pod \"maas-api-e2e-shared-a-38da25-64565dd698-k5fdh\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:10.480521 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.480524    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76848b08-90c1-4723-85c5-3a876bb02b0e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-38da25-64565dd698-k5fdh\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:10.480803 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.480672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28thv\" (UniqueName: \"kubernetes.io/projected/76848b08-90c1-4723-85c5-3a876bb02b0e-kube-api-access-28thv\") pod \"maas-api-e2e-shared-a-38da25-64565dd698-k5fdh\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:10.582194 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.582156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28thv\" (UniqueName: \"kubernetes.io/projected/76848b08-90c1-4723-85c5-3a876bb02b0e-kube-api-access-28thv\") pod \"maas-api-e2e-shared-a-38da25-64565dd698-k5fdh\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:10.582376 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.582218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76848b08-90c1-4723-85c5-3a876bb02b0e-maas-api-tls\") pod \"maas-api-e2e-shared-a-38da25-64565dd698-k5fdh\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:10.582376 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.582241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76848b08-90c1-4723-85c5-3a876bb02b0e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-38da25-64565dd698-k5fdh\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:10.582879 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.582855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76848b08-90c1-4723-85c5-3a876bb02b0e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-38da25-64565dd698-k5fdh\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:10.584723 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.584699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76848b08-90c1-4723-85c5-3a876bb02b0e-maas-api-tls\") pod \"maas-api-e2e-shared-a-38da25-64565dd698-k5fdh\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:10.592114 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.592091    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28thv\" (UniqueName: \"kubernetes.io/projected/76848b08-90c1-4723-85c5-3a876bb02b0e-kube-api-access-28thv\") pod \"maas-api-e2e-shared-a-38da25-64565dd698-k5fdh\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:10.718144 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.718112    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:10.855198 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:10.855048    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"]
Jul 24 00:52:10.858140 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:52:10.858112    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76848b08_90c1_4723_85c5_3a876bb02b0e.slice/crio-3c30fb315b9938cab6a45459da5d3f3c971854f3e7754646ce5f4f0b4833300c WatchSource:0}: Error finding container 3c30fb315b9938cab6a45459da5d3f3c971854f3e7754646ce5f4f0b4833300c: Status 404 returned error can't find the container with id 3c30fb315b9938cab6a45459da5d3f3c971854f3e7754646ce5f4f0b4833300c
Jul 24 00:52:11.088654 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.088564    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"]
Jul 24 00:52:11.093248 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.093222    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 00:52:11.095510 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.095488    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-38da25-dockercfg-kpkf5\""
Jul 24 00:52:11.095629 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.095524    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-38da25\""
Jul 24 00:52:11.101377 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.101355    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"]
Jul 24 00:52:11.127679 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.127643    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"]
Jul 24 00:52:11.131143 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.131123    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 00:52:11.140372 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.140344    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"]
Jul 24 00:52:11.187627 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.187592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh29z\" (UniqueName: \"kubernetes.io/projected/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-kube-api-access-jh29z\") pod \"payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8\" (UID: \"a1bb8779-2fcd-4929-b49a-d3105bdce1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 00:52:11.187820 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.187635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8\" (UID: \"a1bb8779-2fcd-4929-b49a-d3105bdce1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 00:52:11.288897 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.288868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh29z\" (UniqueName: \"kubernetes.io/projected/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-kube-api-access-jh29z\") pod \"payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8\" (UID: \"a1bb8779-2fcd-4929-b49a-d3105bdce1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 00:52:11.289031 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.288912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvwx2\" (UniqueName: \"kubernetes.io/projected/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-kube-api-access-jvwx2\") pod \"payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l\" (UID: \"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236\") " pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 00:52:11.289031 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.288956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8\" (UID: \"a1bb8779-2fcd-4929-b49a-d3105bdce1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 00:52:11.289139 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.289051    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l\" (UID: \"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236\") " pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 00:52:11.289721 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.289701    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8\" (UID: \"a1bb8779-2fcd-4929-b49a-d3105bdce1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 00:52:11.296611 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.296588    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh29z\" (UniqueName: \"kubernetes.io/projected/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-kube-api-access-jh29z\") pod \"payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8\" (UID: \"a1bb8779-2fcd-4929-b49a-d3105bdce1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 00:52:11.389541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.389454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l\" (UID: \"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236\") " pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 00:52:11.389541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.389523    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvwx2\" (UniqueName: \"kubernetes.io/projected/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-kube-api-access-jvwx2\") pod \"payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l\" (UID: \"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236\") " pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 00:52:11.390200 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.390178    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l\" (UID: \"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236\") " pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 00:52:11.398280 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.398256    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvwx2\" (UniqueName: \"kubernetes.io/projected/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-kube-api-access-jvwx2\") pod \"payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l\" (UID: \"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236\") " pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 00:52:11.405019 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.404998    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 00:52:11.442964 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.442931    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 00:52:11.532497 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.532460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh" event={"ID":"76848b08-90c1-4723-85c5-3a876bb02b0e","Type":"ContainerStarted","Data":"752a05723bf0e6e891c8bdfe89c34bdaf38c6c76def985025bec0db2a68d57f2"}
Jul 24 00:52:11.532715 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.532523    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh" event={"ID":"76848b08-90c1-4723-85c5-3a876bb02b0e","Type":"ContainerStarted","Data":"3c30fb315b9938cab6a45459da5d3f3c971854f3e7754646ce5f4f0b4833300c"}
Jul 24 00:52:11.532715 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.532636    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:11.558150 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.558081    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"]
Jul 24 00:52:11.562511 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.562454    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh" podStartSLOduration=1.154632146 podStartE2EDuration="1.562433888s" totalImagesPullingTime="407.801742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:52:11.561445862 +0000 UTC m=+2312.129262013" watchObservedRunningTime="2026-07-24 00:52:11.562433888 +0000 UTC m=+2312.130250039"
Jul 24 00:52:11.563694 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:52:11.563650    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bb8779_2fcd_4929_b49a_d3105bdce1e2.slice/crio-1f2274143d79c9a798769f01ec79bb4c83c056fdcff7136fad72811e10c23bdb WatchSource:0}: Error finding container 1f2274143d79c9a798769f01ec79bb4c83c056fdcff7136fad72811e10c23bdb: Status 404 returned error can't find the container with id 1f2274143d79c9a798769f01ec79bb4c83c056fdcff7136fad72811e10c23bdb
Jul 24 00:52:11.597603 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:11.597578    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"]
Jul 24 00:52:11.599184 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:52:11.599156    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc9d9b8_5fce_46c6_8e2f_65ac22bb3236.slice/crio-99009c5ffe134514adb914ce63156551ae9eeedf228937b31fc8bbe393126ef5 WatchSource:0}: Error finding container 99009c5ffe134514adb914ce63156551ae9eeedf228937b31fc8bbe393126ef5: Status 404 returned error can't find the container with id 99009c5ffe134514adb914ce63156551ae9eeedf228937b31fc8bbe393126ef5
Jul 24 00:52:12.538768 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:12.538711    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8" event={"ID":"a1bb8779-2fcd-4929-b49a-d3105bdce1e2","Type":"ContainerStarted","Data":"ba664aec1a017a385429bafdf4118c26d071df40967e43b8669d8f6f7ecb4b8b"}
Jul 24 00:52:12.539289 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:12.538859    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 00:52:12.539289 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:12.538888    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8" event={"ID":"a1bb8779-2fcd-4929-b49a-d3105bdce1e2","Type":"ContainerStarted","Data":"1f2274143d79c9a798769f01ec79bb4c83c056fdcff7136fad72811e10c23bdb"}
Jul 24 00:52:12.540518 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:12.540491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l" event={"ID":"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236","Type":"ContainerStarted","Data":"e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985"}
Jul 24 00:52:12.540672 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:12.540542    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l" event={"ID":"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236","Type":"ContainerStarted","Data":"99009c5ffe134514adb914ce63156551ae9eeedf228937b31fc8bbe393126ef5"}
Jul 24 00:52:12.540672 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:12.540592    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 00:52:12.555872 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:12.555803    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8" podStartSLOduration=1.5557893360000001 podStartE2EDuration="1.555789336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:52:12.553884004 +0000 UTC m=+2313.121700157" watchObservedRunningTime="2026-07-24 00:52:12.555789336 +0000 UTC m=+2313.123605486"
Jul 24 00:52:12.569921 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:12.569872    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l" podStartSLOduration=1.569858148 podStartE2EDuration="1.569858148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:52:12.568738787 +0000 UTC m=+2313.136554937" watchObservedRunningTime="2026-07-24 00:52:12.569858148 +0000 UTC m=+2313.137674297"
Jul 24 00:52:17.547572 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:17.547534    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 00:52:18.394843 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.394787    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"]
Jul 24 00:52:18.399355 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.399330    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:18.401662 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.401638    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-44edfc-serving-cert\""
Jul 24 00:52:18.407860 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.407818    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"]
Jul 24 00:52:18.552046 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.552008    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:18.552433 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.552058    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts9g5\" (UniqueName: \"kubernetes.io/projected/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-kube-api-access-ts9g5\") pod \"maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:18.552433 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.552132    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:18.653466 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.653379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:18.653466 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.653422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts9g5\" (UniqueName: \"kubernetes.io/projected/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-kube-api-access-ts9g5\") pod \"maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:18.653466 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.653458    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:18.653700 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:52:18.653532    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-serving-cert: secret "maas-api-e2e-shared-b-44edfc-serving-cert" not found
Jul 24 00:52:18.653700 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:52:18.653614    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-maas-api-tls podName:1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1 nodeName:}" failed. No retries permitted until 2026-07-24 00:52:19.153591904 +0000 UTC m=+2319.721408047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-maas-api-tls") pod "maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6" (UID: "1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1") : secret "maas-api-e2e-shared-b-44edfc-serving-cert" not found
Jul 24 00:52:18.654202 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.654182    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:18.662033 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:18.662006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts9g5\" (UniqueName: \"kubernetes.io/projected/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-kube-api-access-ts9g5\") pod \"maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:19.054492 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.054456    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"]
Jul 24 00:52:19.057981 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.057962    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 00:52:19.060957 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.060930    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-44edfc-dockercfg-jz4dn\""
Jul 24 00:52:19.061082 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.060968    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-44edfc\""
Jul 24 00:52:19.069334 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.069309    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"]
Jul 24 00:52:19.095978 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.095947    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"]
Jul 24 00:52:19.099272 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.099255    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 00:52:19.108286 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.108260    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"]
Jul 24 00:52:19.159199 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.159163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:19.159199 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.159203    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l\" (UID: \"f6f5cef3-5b81-49bc-9c95-ac4771978b7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 00:52:19.159431 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.159246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p7t5\" (UniqueName: \"kubernetes.io/projected/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-kube-api-access-8p7t5\") pod \"payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l\" (UID: \"f6f5cef3-5b81-49bc-9c95-ac4771978b7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 00:52:19.161901 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.161872    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:19.260129 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.260084    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc58066-e31c-452b-acef-b63dda1465f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t\" (UID: \"6dc58066-e31c-452b-acef-b63dda1465f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 00:52:19.260339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.260156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l\" (UID: \"f6f5cef3-5b81-49bc-9c95-ac4771978b7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 00:52:19.260339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.260182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsn24\" (UniqueName: \"kubernetes.io/projected/6dc58066-e31c-452b-acef-b63dda1465f1-kube-api-access-zsn24\") pod \"payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t\" (UID: \"6dc58066-e31c-452b-acef-b63dda1465f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 00:52:19.260339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.260239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p7t5\" (UniqueName: \"kubernetes.io/projected/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-kube-api-access-8p7t5\") pod \"payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l\" (UID: \"f6f5cef3-5b81-49bc-9c95-ac4771978b7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 00:52:19.260890 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.260868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l\" (UID: \"f6f5cef3-5b81-49bc-9c95-ac4771978b7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 00:52:19.268657 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.268629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p7t5\" (UniqueName: \"kubernetes.io/projected/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-kube-api-access-8p7t5\") pod \"payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l\" (UID: \"f6f5cef3-5b81-49bc-9c95-ac4771978b7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 00:52:19.311866 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.311765    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:19.360982 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.360943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc58066-e31c-452b-acef-b63dda1465f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t\" (UID: \"6dc58066-e31c-452b-acef-b63dda1465f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 00:52:19.361170 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.361014    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsn24\" (UniqueName: \"kubernetes.io/projected/6dc58066-e31c-452b-acef-b63dda1465f1-kube-api-access-zsn24\") pod \"payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t\" (UID: \"6dc58066-e31c-452b-acef-b63dda1465f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 00:52:19.361814 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.361786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc58066-e31c-452b-acef-b63dda1465f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t\" (UID: \"6dc58066-e31c-452b-acef-b63dda1465f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 00:52:19.369508 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.369478    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 00:52:19.370104 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.370080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsn24\" (UniqueName: \"kubernetes.io/projected/6dc58066-e31c-452b-acef-b63dda1465f1-kube-api-access-zsn24\") pod \"payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t\" (UID: \"6dc58066-e31c-452b-acef-b63dda1465f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 00:52:19.410220 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.410184    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 00:52:19.465367 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.465339    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"]
Jul 24 00:52:19.466641 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:52:19.466608    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac0cdc3_eb7b_4582_9bb7_d60998c1d5a1.slice/crio-92782ef7ad5e2d07960cef1abe4fc6d9f461ce41fcfe103a84bb56fe9a8486d1 WatchSource:0}: Error finding container 92782ef7ad5e2d07960cef1abe4fc6d9f461ce41fcfe103a84bb56fe9a8486d1: Status 404 returned error can't find the container with id 92782ef7ad5e2d07960cef1abe4fc6d9f461ce41fcfe103a84bb56fe9a8486d1
Jul 24 00:52:19.531506 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.531471    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"]
Jul 24 00:52:19.535859 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:52:19.535808    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f5cef3_5b81_49bc_9c95_ac4771978b7f.slice/crio-cbae8e2e0e0b8cf1ddf201bf6bad3c8cb9b58230e6a013a86113be5be7d3b43c WatchSource:0}: Error finding container cbae8e2e0e0b8cf1ddf201bf6bad3c8cb9b58230e6a013a86113be5be7d3b43c: Status 404 returned error can't find the container with id cbae8e2e0e0b8cf1ddf201bf6bad3c8cb9b58230e6a013a86113be5be7d3b43c
Jul 24 00:52:19.576008 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.575943    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"]
Jul 24 00:52:19.578813 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:52:19.578782    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc58066_e31c_452b_acef_b63dda1465f1.slice/crio-1c45814efcaa02307a5809ac988f04ba1f9764d3d35f9e513bbd182919ae05a7 WatchSource:0}: Error finding container 1c45814efcaa02307a5809ac988f04ba1f9764d3d35f9e513bbd182919ae05a7: Status 404 returned error can't find the container with id 1c45814efcaa02307a5809ac988f04ba1f9764d3d35f9e513bbd182919ae05a7
Jul 24 00:52:19.578956 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.578865    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l" event={"ID":"f6f5cef3-5b81-49bc-9c95-ac4771978b7f","Type":"ContainerStarted","Data":"cbae8e2e0e0b8cf1ddf201bf6bad3c8cb9b58230e6a013a86113be5be7d3b43c"}
Jul 24 00:52:19.580368 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:19.580339    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6" event={"ID":"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1","Type":"ContainerStarted","Data":"92782ef7ad5e2d07960cef1abe4fc6d9f461ce41fcfe103a84bb56fe9a8486d1"}
Jul 24 00:52:20.591326 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:20.591289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l" event={"ID":"f6f5cef3-5b81-49bc-9c95-ac4771978b7f","Type":"ContainerStarted","Data":"73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88"}
Jul 24 00:52:20.591783 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:20.591422    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 00:52:20.592931 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:20.592905    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6" event={"ID":"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1","Type":"ContainerStarted","Data":"4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa"}
Jul 24 00:52:20.593012 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:20.592976    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:20.594561 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:20.594540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t" event={"ID":"6dc58066-e31c-452b-acef-b63dda1465f1","Type":"ContainerStarted","Data":"a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f"}
Jul 24 00:52:20.594686 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:20.594574    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t" event={"ID":"6dc58066-e31c-452b-acef-b63dda1465f1","Type":"ContainerStarted","Data":"1c45814efcaa02307a5809ac988f04ba1f9764d3d35f9e513bbd182919ae05a7"}
Jul 24 00:52:20.594686 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:20.594602    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 00:52:20.608479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:20.608439    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l" podStartSLOduration=1.6084279430000001 podStartE2EDuration="1.608427943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:52:20.606178797 +0000 UTC m=+2321.173994947" watchObservedRunningTime="2026-07-24 00:52:20.608427943 +0000 UTC m=+2321.176244163"
Jul 24 00:52:20.623822 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:20.623780    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t" podStartSLOduration=1.6237669289999999 podStartE2EDuration="1.623766929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:52:20.620964193 +0000 UTC m=+2321.188780345" watchObservedRunningTime="2026-07-24 00:52:20.623766929 +0000 UTC m=+2321.191583147"
Jul 24 00:52:20.637782 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:20.637740    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6" podStartSLOduration=2.290608167 podStartE2EDuration="2.637725316s" totalImagesPullingTime="347.117149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:52:20.63754212 +0000 UTC m=+2321.205358274" watchObservedRunningTime="2026-07-24 00:52:20.637725316 +0000 UTC m=+2321.205541466"
Jul 24 00:52:23.547953 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:23.547921    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 00:52:23.548666 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:23.548006    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 00:52:26.605394 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:26.605360    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 00:52:31.601397 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:31.601351    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t" podUID="6dc58066-e31c-452b-acef-b63dda1465f1" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.78:9004: connect: connection refused"
Jul 24 00:52:31.601983 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:31.601963    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 00:52:39.280243 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.280166    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"]
Jul 24 00:52:39.285121 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.285096    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.288543 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.288498    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-38da25\"/\"default-dockercfg-qsk9j\""
Jul 24 00:52:39.288543 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.288527    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-38da25\"/\"auth-test-model-38da25-kserve-self-signed-certs\""
Jul 24 00:52:39.288543 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.288498    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-38da25\"/\"openshift-service-ca.crt\""
Jul 24 00:52:39.288902 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.288538    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-38da25\"/\"kube-root-ca.crt\""
Jul 24 00:52:39.291808 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.291782    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"]
Jul 24 00:52:39.325423 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.325387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-model-cache\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.325580 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.325459    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-home\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.325580 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.325490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-dshm\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.325580 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.325509    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgrq8\" (UniqueName: \"kubernetes.io/projected/8ab6c652-4db8-4e45-8c91-74f6813bda21-kube-api-access-mgrq8\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.325580 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.325568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab6c652-4db8-4e45-8c91-74f6813bda21-tls-certs\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.325725 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.325629    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-tmp-dir\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.426654 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.426612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab6c652-4db8-4e45-8c91-74f6813bda21-tls-certs\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.426865 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.426666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-tmp-dir\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.426865 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.426710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-model-cache\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.426865 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.426736    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-home\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.426865 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.426755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-dshm\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.426865 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.426774    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrq8\" (UniqueName: \"kubernetes.io/projected/8ab6c652-4db8-4e45-8c91-74f6813bda21-kube-api-access-mgrq8\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.427179 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.427155    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-tmp-dir\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.427258 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.427204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-home\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.427258 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.427247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-model-cache\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.429045 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.429025    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-dshm\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.429227 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.429208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab6c652-4db8-4e45-8c91-74f6813bda21-tls-certs\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.435020 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.434991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrq8\" (UniqueName: \"kubernetes.io/projected/8ab6c652-4db8-4e45-8c91-74f6813bda21-kube-api-access-mgrq8\") pod \"auth-test-model-38da25-kserve-67cd5cb77f-wmnhb\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") " pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.596007 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.595918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:39.743133 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:39.742935    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"]
Jul 24 00:52:39.746155 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:52:39.746113    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ab6c652_4db8_4e45_8c91_74f6813bda21.slice/crio-e60f370987d131f4710af81eb3f2032afe6b4df8f003e9c314b3a2f641c0131f WatchSource:0}: Error finding container e60f370987d131f4710af81eb3f2032afe6b4df8f003e9c314b3a2f641c0131f: Status 404 returned error can't find the container with id e60f370987d131f4710af81eb3f2032afe6b4df8f003e9c314b3a2f641c0131f
Jul 24 00:52:40.696028 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:40.695929    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb" event={"ID":"8ab6c652-4db8-4e45-8c91-74f6813bda21","Type":"ContainerStarted","Data":"3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f"}
Jul 24 00:52:40.696028 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:40.695978    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb" event={"ID":"8ab6c652-4db8-4e45-8c91-74f6813bda21","Type":"ContainerStarted","Data":"e60f370987d131f4710af81eb3f2032afe6b4df8f003e9c314b3a2f641c0131f"}
Jul 24 00:52:40.713048 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:40.712966    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb" podStartSLOduration=1.712941465 podStartE2EDuration="1.712941465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:52:40.711911719 +0000 UTC m=+2341.279727870" watchObservedRunningTime="2026-07-24 00:52:40.712941465 +0000 UTC m=+2341.280757617"
Jul 24 00:52:41.600607 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:41.600576    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 00:52:49.596935 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:49.596892    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:49.597338 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:49.596950    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:49.610945 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:49.610917    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:52:49.751361 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:52:49.751333    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:55:09.172108 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:09.172071    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"]
Jul 24 00:55:24.191681 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.191634    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb" podUID="8ab6c652-4db8-4e45-8c91-74f6813bda21" containerName="main" containerID="cri-o://3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f" gracePeriod=45
Jul 24 00:55:24.448972 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.448952    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:55:24.507063 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.507026    2560 generic.go:356] "Generic (PLEG): container finished" podID="8ab6c652-4db8-4e45-8c91-74f6813bda21" containerID="3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f" exitCode=0
Jul 24 00:55:24.507228 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.507069    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"
Jul 24 00:55:24.507228 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.507111    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb" event={"ID":"8ab6c652-4db8-4e45-8c91-74f6813bda21","Type":"ContainerDied","Data":"3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f"}
Jul 24 00:55:24.507228 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.507156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb" event={"ID":"8ab6c652-4db8-4e45-8c91-74f6813bda21","Type":"ContainerDied","Data":"e60f370987d131f4710af81eb3f2032afe6b4df8f003e9c314b3a2f641c0131f"}
Jul 24 00:55:24.507228 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.507177    2560 scope.go:117] "RemoveContainer" containerID="3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f"
Jul 24 00:55:24.515617 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.515597    2560 scope.go:117] "RemoveContainer" containerID="3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f"
Jul 24 00:55:24.515904 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:55:24.515882    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f\": container with ID starting with 3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f not found: ID does not exist" containerID="3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f"
Jul 24 00:55:24.515976 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.515914    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f"} err="failed to get container status \"3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f\": rpc error: code = NotFound desc = could not find container \"3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f\": container with ID starting with 3411d8d46c719b671c41f0901711fa4cb695aefb0f4f1eaf27706373b0d5af1f not found: ID does not exist"
Jul 24 00:55:24.569364 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.569338    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-dshm\") pod \"8ab6c652-4db8-4e45-8c91-74f6813bda21\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") "
Jul 24 00:55:24.569513 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.569402    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab6c652-4db8-4e45-8c91-74f6813bda21-tls-certs\") pod \"8ab6c652-4db8-4e45-8c91-74f6813bda21\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") "
Jul 24 00:55:24.569513 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.569452    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgrq8\" (UniqueName: \"kubernetes.io/projected/8ab6c652-4db8-4e45-8c91-74f6813bda21-kube-api-access-mgrq8\") pod \"8ab6c652-4db8-4e45-8c91-74f6813bda21\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") "
Jul 24 00:55:24.569513 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.569488    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-model-cache\") pod \"8ab6c652-4db8-4e45-8c91-74f6813bda21\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") "
Jul 24 00:55:24.569635 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.569529    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-tmp-dir\") pod \"8ab6c652-4db8-4e45-8c91-74f6813bda21\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") "
Jul 24 00:55:24.569635 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.569557    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-home\") pod \"8ab6c652-4db8-4e45-8c91-74f6813bda21\" (UID: \"8ab6c652-4db8-4e45-8c91-74f6813bda21\") "
Jul 24 00:55:24.569826 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.569800    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ab6c652-4db8-4e45-8c91-74f6813bda21" (UID: "8ab6c652-4db8-4e45-8c91-74f6813bda21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:24.569928 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.569815    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ab6c652-4db8-4e45-8c91-74f6813bda21" (UID: "8ab6c652-4db8-4e45-8c91-74f6813bda21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:24.569969 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.569921    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-home" (OuterVolumeSpecName: "home") pod "8ab6c652-4db8-4e45-8c91-74f6813bda21" (UID: "8ab6c652-4db8-4e45-8c91-74f6813bda21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:24.571802 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.571780    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab6c652-4db8-4e45-8c91-74f6813bda21-kube-api-access-mgrq8" (OuterVolumeSpecName: "kube-api-access-mgrq8") pod "8ab6c652-4db8-4e45-8c91-74f6813bda21" (UID: "8ab6c652-4db8-4e45-8c91-74f6813bda21"). InnerVolumeSpecName "kube-api-access-mgrq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:55:24.571802 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.571796    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-dshm" (OuterVolumeSpecName: "dshm") pod "8ab6c652-4db8-4e45-8c91-74f6813bda21" (UID: "8ab6c652-4db8-4e45-8c91-74f6813bda21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:55:24.572027 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.571779    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab6c652-4db8-4e45-8c91-74f6813bda21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ab6c652-4db8-4e45-8c91-74f6813bda21" (UID: "8ab6c652-4db8-4e45-8c91-74f6813bda21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:55:24.670723 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.670688    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-dshm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:55:24.670723 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.670717    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab6c652-4db8-4e45-8c91-74f6813bda21-tls-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:55:24.670723 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.670729    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgrq8\" (UniqueName: \"kubernetes.io/projected/8ab6c652-4db8-4e45-8c91-74f6813bda21-kube-api-access-mgrq8\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:55:24.671066 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.670737    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-model-cache\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:55:24.671066 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.670745    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-tmp-dir\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:55:24.671066 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.670753    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab6c652-4db8-4e45-8c91-74f6813bda21-home\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:55:24.830761 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.830720    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"]
Jul 24 00:55:24.832928 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:24.832904    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-38da25/auth-test-model-38da25-kserve-67cd5cb77f-wmnhb"]
Jul 24 00:55:25.968070 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:55:25.968033    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab6c652-4db8-4e45-8c91-74f6813bda21" path="/var/lib/kubelet/pods/8ab6c652-4db8-4e45-8c91-74f6813bda21/volumes"
Jul 24 00:56:32.503849 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.503788    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"]
Jul 24 00:56:32.506435 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.504366    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab6c652-4db8-4e45-8c91-74f6813bda21" containerName="main"
Jul 24 00:56:32.506435 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.504384    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab6c652-4db8-4e45-8c91-74f6813bda21" containerName="main"
Jul 24 00:56:32.506435 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.504456    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab6c652-4db8-4e45-8c91-74f6813bda21" containerName="main"
Jul 24 00:56:32.507343 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.507323    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.510758 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.510735    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-38da25\"/\"rate-test-model-38da25-kserve-self-signed-certs\""
Jul 24 00:56:32.510907 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.510734    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-38da25\"/\"kube-root-ca.crt\""
Jul 24 00:56:32.511113 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.511094    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-38da25\"/\"openshift-service-ca.crt\""
Jul 24 00:56:32.511177 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.511125    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-38da25\"/\"default-dockercfg-qsk9j\""
Jul 24 00:56:32.522297 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.518949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"]
Jul 24 00:56:32.690220 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.690178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-tmp-dir\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.690432 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.690325    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-dshm\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.690432 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.690370    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4660c2a-0662-440a-97ed-253f5341ef9e-tls-certs\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.690432 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.690424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-home\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.690585 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.690450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt65z\" (UniqueName: \"kubernetes.io/projected/d4660c2a-0662-440a-97ed-253f5341ef9e-kube-api-access-dt65z\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.690585 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.690554    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-model-cache\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.791200 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.791100    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-dshm\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.791200 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.791155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4660c2a-0662-440a-97ed-253f5341ef9e-tls-certs\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.791426 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.791288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-home\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.791426 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.791352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt65z\" (UniqueName: \"kubernetes.io/projected/d4660c2a-0662-440a-97ed-253f5341ef9e-kube-api-access-dt65z\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.791544 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.791473    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-model-cache\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.791544 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.791506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-tmp-dir\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.791677 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.791648    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-home\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.791820 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.791800    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-tmp-dir\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.791888 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.791815    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-model-cache\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.793557 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.793529    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-dshm\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.793755 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.793740    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4660c2a-0662-440a-97ed-253f5341ef9e-tls-certs\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.800444 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.800421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt65z\" (UniqueName: \"kubernetes.io/projected/d4660c2a-0662-440a-97ed-253f5341ef9e-kube-api-access-dt65z\") pod \"rate-test-model-38da25-kserve-88b4fd4d7-ncf8r\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") " pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.823260 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.823233    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:32.958393 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:32.958366    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"]
Jul 24 00:56:32.960582 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:56:32.960550    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4660c2a_0662_440a_97ed_253f5341ef9e.slice/crio-30f797dc900f99daf55fb5bd778bb2c2bf02d36ac9f85829fc4648f6e3db00d5 WatchSource:0}: Error finding container 30f797dc900f99daf55fb5bd778bb2c2bf02d36ac9f85829fc4648f6e3db00d5: Status 404 returned error can't find the container with id 30f797dc900f99daf55fb5bd778bb2c2bf02d36ac9f85829fc4648f6e3db00d5
Jul 24 00:56:33.853109 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:33.853072    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r" event={"ID":"d4660c2a-0662-440a-97ed-253f5341ef9e","Type":"ContainerStarted","Data":"3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292"}
Jul 24 00:56:33.853109 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:33.853114    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r" event={"ID":"d4660c2a-0662-440a-97ed-253f5341ef9e","Type":"ContainerStarted","Data":"30f797dc900f99daf55fb5bd778bb2c2bf02d36ac9f85829fc4648f6e3db00d5"}
Jul 24 00:56:33.870233 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:33.870180    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r" podStartSLOduration=1.870166129 podStartE2EDuration="1.870166129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:56:33.867940861 +0000 UTC m=+2574.435757010" watchObservedRunningTime="2026-07-24 00:56:33.870166129 +0000 UTC m=+2574.437982279"
Jul 24 00:56:40.270905 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:40.270871    2560 scope.go:117] "RemoveContainer" containerID="6257134ccd2942113474bd4f50c9b9e669d8c342994803d5be0aeda2c684ef9f"
Jul 24 00:56:42.824159 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:42.824120    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:42.824553 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:42.824177    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:42.836983 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:42.836957    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:56:42.910679 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:56:42.910646    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:57:40.289625 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:57:40.289593    2560 scope.go:117] "RemoveContainer" containerID="5b875feaf7c3b98012eef29288a183094ec1b07395b55ff04e2a4d7f631caf1f"
Jul 24 00:57:54.973036 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:57:54.973002    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"]
Jul 24 00:58:02.702005 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:02.701967    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7358-140013704328944"]
Jul 24 00:58:02.705713 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:02.705691    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013704328944"
Jul 24 00:58:02.708012 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:02.707993    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wmrlv\""
Jul 24 00:58:02.710966 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:02.710942    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7358-140013704328944"]
Jul 24 00:58:02.883523 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:02.883483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh5tg\" (UniqueName: \"kubernetes.io/projected/7d3e6bdc-953f-4f59-8604-bd75a408cfb8-kube-api-access-sh5tg\") pod \"test-curl-7358-140013704328944\" (UID: \"7d3e6bdc-953f-4f59-8604-bd75a408cfb8\") " pod="opendatahub/test-curl-7358-140013704328944"
Jul 24 00:58:02.987654 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:02.987554    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh5tg\" (UniqueName: \"kubernetes.io/projected/7d3e6bdc-953f-4f59-8604-bd75a408cfb8-kube-api-access-sh5tg\") pod \"test-curl-7358-140013704328944\" (UID: \"7d3e6bdc-953f-4f59-8604-bd75a408cfb8\") " pod="opendatahub/test-curl-7358-140013704328944"
Jul 24 00:58:02.996299 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:02.996274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh5tg\" (UniqueName: \"kubernetes.io/projected/7d3e6bdc-953f-4f59-8604-bd75a408cfb8-kube-api-access-sh5tg\") pod \"test-curl-7358-140013704328944\" (UID: \"7d3e6bdc-953f-4f59-8604-bd75a408cfb8\") " pod="opendatahub/test-curl-7358-140013704328944"
Jul 24 00:58:03.016988 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:03.016944    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013704328944"
Jul 24 00:58:03.150288 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:03.150231    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7358-140013704328944"]
Jul 24 00:58:03.152700 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:58:03.152661    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d3e6bdc_953f_4f59_8604_bd75a408cfb8.slice/crio-e399873c701fc0d3a2c5fda12e9e9d781ea50339150be32355a337e3a05341c6 WatchSource:0}: Error finding container e399873c701fc0d3a2c5fda12e9e9d781ea50339150be32355a337e3a05341c6: Status 404 returned error can't find the container with id e399873c701fc0d3a2c5fda12e9e9d781ea50339150be32355a337e3a05341c6
Jul 24 00:58:03.154077 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:03.154059    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 00:58:03.289824 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:03.289745    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7358-140013704328944" event={"ID":"7d3e6bdc-953f-4f59-8604-bd75a408cfb8","Type":"ContainerStarted","Data":"e399873c701fc0d3a2c5fda12e9e9d781ea50339150be32355a337e3a05341c6"}
Jul 24 00:58:04.297331 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:04.297290    2560 generic.go:356] "Generic (PLEG): container finished" podID="7d3e6bdc-953f-4f59-8604-bd75a408cfb8" containerID="9f4e8e4467a18bd16e784a49cae60855924ed496900af48b788e4169757e0401" exitCode=0
Jul 24 00:58:04.297331 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:04.297326    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7358-140013704328944" event={"ID":"7d3e6bdc-953f-4f59-8604-bd75a408cfb8","Type":"ContainerDied","Data":"9f4e8e4467a18bd16e784a49cae60855924ed496900af48b788e4169757e0401"}
Jul 24 00:58:05.437520 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:05.437491    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013704328944"
Jul 24 00:58:05.470481 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:05.470456    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7358-140013704328944_7d3e6bdc-953f-4f59-8604-bd75a408cfb8/test-curl-7358-140013704328944/0.log"
Jul 24 00:58:05.491315 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:05.491281    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7358-140013704328944"]
Jul 24 00:58:05.496688 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:05.496664    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7358-140013704328944"]
Jul 24 00:58:05.506122 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:05.506092    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh5tg\" (UniqueName: \"kubernetes.io/projected/7d3e6bdc-953f-4f59-8604-bd75a408cfb8-kube-api-access-sh5tg\") pod \"7d3e6bdc-953f-4f59-8604-bd75a408cfb8\" (UID: \"7d3e6bdc-953f-4f59-8604-bd75a408cfb8\") "
Jul 24 00:58:05.508186 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:05.508150    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3e6bdc-953f-4f59-8604-bd75a408cfb8-kube-api-access-sh5tg" (OuterVolumeSpecName: "kube-api-access-sh5tg") pod "7d3e6bdc-953f-4f59-8604-bd75a408cfb8" (UID: "7d3e6bdc-953f-4f59-8604-bd75a408cfb8"). InnerVolumeSpecName "kube-api-access-sh5tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:58:05.607694 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:05.607609    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh5tg\" (UniqueName: \"kubernetes.io/projected/7d3e6bdc-953f-4f59-8604-bd75a408cfb8-kube-api-access-sh5tg\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:58:05.967012 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:05.966977    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d3e6bdc-953f-4f59-8604-bd75a408cfb8" path="/var/lib/kubelet/pods/7d3e6bdc-953f-4f59-8604-bd75a408cfb8/volumes"
Jul 24 00:58:06.307944 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:06.307859    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013704328944"
Jul 24 00:58:06.307944 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:06.307898    2560 scope.go:117] "RemoveContainer" containerID="9f4e8e4467a18bd16e784a49cae60855924ed496900af48b788e4169757e0401"
Jul 24 00:58:09.992107 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:09.992063    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r" podUID="d4660c2a-0662-440a-97ed-253f5341ef9e" containerName="main" containerID="cri-o://3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292" gracePeriod=45
Jul 24 00:58:10.248399 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.248337    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:58:10.329479 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.329444    2560 generic.go:356] "Generic (PLEG): container finished" podID="d4660c2a-0662-440a-97ed-253f5341ef9e" containerID="3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292" exitCode=0
Jul 24 00:58:10.329668 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.329493    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"
Jul 24 00:58:10.329668 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.329496    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r" event={"ID":"d4660c2a-0662-440a-97ed-253f5341ef9e","Type":"ContainerDied","Data":"3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292"}
Jul 24 00:58:10.329668 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.329601    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r" event={"ID":"d4660c2a-0662-440a-97ed-253f5341ef9e","Type":"ContainerDied","Data":"30f797dc900f99daf55fb5bd778bb2c2bf02d36ac9f85829fc4648f6e3db00d5"}
Jul 24 00:58:10.329668 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.329625    2560 scope.go:117] "RemoveContainer" containerID="3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292"
Jul 24 00:58:10.338002 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.337982    2560 scope.go:117] "RemoveContainer" containerID="3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292"
Jul 24 00:58:10.338288 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:58:10.338268    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292\": container with ID starting with 3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292 not found: ID does not exist" containerID="3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292"
Jul 24 00:58:10.338339 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.338298    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292"} err="failed to get container status \"3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292\": rpc error: code = NotFound desc = could not find container \"3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292\": container with ID starting with 3fb13155370676aa50dfc0e7dbc100962e4fb18ab0ca1076ad341b1b71e5a292 not found: ID does not exist"
Jul 24 00:58:10.348179 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.348156    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-tmp-dir\") pod \"d4660c2a-0662-440a-97ed-253f5341ef9e\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") "
Jul 24 00:58:10.348276 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.348213    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt65z\" (UniqueName: \"kubernetes.io/projected/d4660c2a-0662-440a-97ed-253f5341ef9e-kube-api-access-dt65z\") pod \"d4660c2a-0662-440a-97ed-253f5341ef9e\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") "
Jul 24 00:58:10.348276 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.348263    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-home\") pod \"d4660c2a-0662-440a-97ed-253f5341ef9e\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") "
Jul 24 00:58:10.348374 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.348282    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-model-cache\") pod \"d4660c2a-0662-440a-97ed-253f5341ef9e\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") "
Jul 24 00:58:10.348374 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.348305    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-dshm\") pod \"d4660c2a-0662-440a-97ed-253f5341ef9e\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") "
Jul 24 00:58:10.348374 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.348328    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4660c2a-0662-440a-97ed-253f5341ef9e-tls-certs\") pod \"d4660c2a-0662-440a-97ed-253f5341ef9e\" (UID: \"d4660c2a-0662-440a-97ed-253f5341ef9e\") "
Jul 24 00:58:10.348532 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.348433    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4660c2a-0662-440a-97ed-253f5341ef9e" (UID: "d4660c2a-0662-440a-97ed-253f5341ef9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:58:10.348591 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.348529    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-home" (OuterVolumeSpecName: "home") pod "d4660c2a-0662-440a-97ed-253f5341ef9e" (UID: "d4660c2a-0662-440a-97ed-253f5341ef9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:58:10.348591 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.348549    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4660c2a-0662-440a-97ed-253f5341ef9e" (UID: "d4660c2a-0662-440a-97ed-253f5341ef9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:58:10.348736 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.348596    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-tmp-dir\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:58:10.348736 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.348615    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-home\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:58:10.350262 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.350242    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4660c2a-0662-440a-97ed-253f5341ef9e-kube-api-access-dt65z" (OuterVolumeSpecName: "kube-api-access-dt65z") pod "d4660c2a-0662-440a-97ed-253f5341ef9e" (UID: "d4660c2a-0662-440a-97ed-253f5341ef9e"). InnerVolumeSpecName "kube-api-access-dt65z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:58:10.350372 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.350349    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-dshm" (OuterVolumeSpecName: "dshm") pod "d4660c2a-0662-440a-97ed-253f5341ef9e" (UID: "d4660c2a-0662-440a-97ed-253f5341ef9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 00:58:10.350462 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.350440    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4660c2a-0662-440a-97ed-253f5341ef9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4660c2a-0662-440a-97ed-253f5341ef9e" (UID: "d4660c2a-0662-440a-97ed-253f5341ef9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 00:58:10.449754 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.449724    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt65z\" (UniqueName: \"kubernetes.io/projected/d4660c2a-0662-440a-97ed-253f5341ef9e-kube-api-access-dt65z\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:58:10.449754 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.449752    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-model-cache\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:58:10.449754 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.449761    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4660c2a-0662-440a-97ed-253f5341ef9e-dshm\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:58:10.449754 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.449769    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4660c2a-0662-440a-97ed-253f5341ef9e-tls-certs\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:58:10.651634 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.651602    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"]
Jul 24 00:58:10.658221 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:10.658193    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-38da25/rate-test-model-38da25-kserve-88b4fd4d7-ncf8r"]
Jul 24 00:58:11.762338 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.762294    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7358-140013700301136"]
Jul 24 00:58:11.762907 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.762730    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4660c2a-0662-440a-97ed-253f5341ef9e" containerName="main"
Jul 24 00:58:11.762907 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.762743    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4660c2a-0662-440a-97ed-253f5341ef9e" containerName="main"
Jul 24 00:58:11.762907 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.762758    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d3e6bdc-953f-4f59-8604-bd75a408cfb8" containerName="test-curl-7358-140013704328944"
Jul 24 00:58:11.762907 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.762764    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d3e6bdc-953f-4f59-8604-bd75a408cfb8" containerName="test-curl-7358-140013704328944"
Jul 24 00:58:11.762907 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.762862    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4660c2a-0662-440a-97ed-253f5341ef9e" containerName="main"
Jul 24 00:58:11.762907 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.762876    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d3e6bdc-953f-4f59-8604-bd75a408cfb8" containerName="test-curl-7358-140013704328944"
Jul 24 00:58:11.767128 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.767108    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700301136"
Jul 24 00:58:11.769485 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.769465    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wmrlv\""
Jul 24 00:58:11.770733 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.770709    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7358-140013700301136"]
Jul 24 00:58:11.863925 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.863882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhvv\" (UniqueName: \"kubernetes.io/projected/ac90606e-8a53-404b-ab25-a3f302e581d2-kube-api-access-wdhvv\") pod \"test-curl-7358-140013700301136\" (UID: \"ac90606e-8a53-404b-ab25-a3f302e581d2\") " pod="opendatahub/test-curl-7358-140013700301136"
Jul 24 00:58:11.965141 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.965111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhvv\" (UniqueName: \"kubernetes.io/projected/ac90606e-8a53-404b-ab25-a3f302e581d2-kube-api-access-wdhvv\") pod \"test-curl-7358-140013700301136\" (UID: \"ac90606e-8a53-404b-ab25-a3f302e581d2\") " pod="opendatahub/test-curl-7358-140013700301136"
Jul 24 00:58:11.967696 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.967668    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4660c2a-0662-440a-97ed-253f5341ef9e" path="/var/lib/kubelet/pods/d4660c2a-0662-440a-97ed-253f5341ef9e/volumes"
Jul 24 00:58:11.972808 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:11.972788    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhvv\" (UniqueName: \"kubernetes.io/projected/ac90606e-8a53-404b-ab25-a3f302e581d2-kube-api-access-wdhvv\") pod \"test-curl-7358-140013700301136\" (UID: \"ac90606e-8a53-404b-ab25-a3f302e581d2\") " pod="opendatahub/test-curl-7358-140013700301136"
Jul 24 00:58:12.078642 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:12.078555    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700301136"
Jul 24 00:58:12.210530 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:12.210469    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7358-140013700301136"]
Jul 24 00:58:12.213114 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:58:12.213085    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac90606e_8a53_404b_ab25_a3f302e581d2.slice/crio-5d9b58d973efd030d72f3c00f8f6eb9120a389edb4777e12b403b3b56a11f01d WatchSource:0}: Error finding container 5d9b58d973efd030d72f3c00f8f6eb9120a389edb4777e12b403b3b56a11f01d: Status 404 returned error can't find the container with id 5d9b58d973efd030d72f3c00f8f6eb9120a389edb4777e12b403b3b56a11f01d
Jul 24 00:58:12.340641 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:12.340558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7358-140013700301136" event={"ID":"ac90606e-8a53-404b-ab25-a3f302e581d2","Type":"ContainerStarted","Data":"5d9b58d973efd030d72f3c00f8f6eb9120a389edb4777e12b403b3b56a11f01d"}
Jul 24 00:58:13.347405 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:13.347312    2560 generic.go:356] "Generic (PLEG): container finished" podID="ac90606e-8a53-404b-ab25-a3f302e581d2" containerID="3da53131b0dd7ffb0fec7b0c1e6dac36a67185606f592ca399eea43c3324ffa1" exitCode=0
Jul 24 00:58:13.347405 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:13.347376    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7358-140013700301136" event={"ID":"ac90606e-8a53-404b-ab25-a3f302e581d2","Type":"ContainerDied","Data":"3da53131b0dd7ffb0fec7b0c1e6dac36a67185606f592ca399eea43c3324ffa1"}
Jul 24 00:58:14.495736 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:14.495707    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700301136"
Jul 24 00:58:14.514714 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:14.514684    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7358-140013700301136_ac90606e-8a53-404b-ab25-a3f302e581d2/test-curl-7358-140013700301136/0.log"
Jul 24 00:58:14.539476 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:14.539450    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7358-140013700301136"]
Jul 24 00:58:14.542252 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:14.542227    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7358-140013700301136"]
Jul 24 00:58:14.586510 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:14.586478    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdhvv\" (UniqueName: \"kubernetes.io/projected/ac90606e-8a53-404b-ab25-a3f302e581d2-kube-api-access-wdhvv\") pod \"ac90606e-8a53-404b-ab25-a3f302e581d2\" (UID: \"ac90606e-8a53-404b-ab25-a3f302e581d2\") "
Jul 24 00:58:14.588541 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:14.588515    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac90606e-8a53-404b-ab25-a3f302e581d2-kube-api-access-wdhvv" (OuterVolumeSpecName: "kube-api-access-wdhvv") pod "ac90606e-8a53-404b-ab25-a3f302e581d2" (UID: "ac90606e-8a53-404b-ab25-a3f302e581d2"). InnerVolumeSpecName "kube-api-access-wdhvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 00:58:14.687809 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:14.687715    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdhvv\" (UniqueName: \"kubernetes.io/projected/ac90606e-8a53-404b-ab25-a3f302e581d2-kube-api-access-wdhvv\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 00:58:15.363680 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:15.363645    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d9b58d973efd030d72f3c00f8f6eb9120a389edb4777e12b403b3b56a11f01d"
Jul 24 00:58:15.363904 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:15.363723    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7358-140013700301136"
Jul 24 00:58:15.374804 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:58:15.374770    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7358-140013700301136\" is forbidden: User \"system:node:ip-10-0-135-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-176.ec2.internal' and this object" podUID="ac90606e-8a53-404b-ab25-a3f302e581d2" pod="opendatahub/test-curl-7358-140013700301136"
Jul 24 00:58:15.967189 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:15.967152    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac90606e-8a53-404b-ab25-a3f302e581d2" path="/var/lib/kubelet/pods/ac90606e-8a53-404b-ab25-a3f302e581d2/volumes"
Jul 24 00:58:18.604702 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.604662    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"]
Jul 24 00:58:18.605116 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.605100    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac90606e-8a53-404b-ab25-a3f302e581d2" containerName="test-curl-7358-140013700301136"
Jul 24 00:58:18.605116 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.605116    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac90606e-8a53-404b-ab25-a3f302e581d2" containerName="test-curl-7358-140013700301136"
Jul 24 00:58:18.605240 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.605228    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac90606e-8a53-404b-ab25-a3f302e581d2" containerName="test-curl-7358-140013700301136"
Jul 24 00:58:18.608452 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.608431    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:18.611023 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.610999    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a3cf0e-serving-cert\""
Jul 24 00:58:18.619631 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.619611    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"]
Jul 24 00:58:18.719217 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.719178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrl9\" (UniqueName: \"kubernetes.io/projected/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-kube-api-access-4jrl9\") pod \"maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:18.719394 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.719232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:18.719394 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.719333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:18.820620 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.820582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:18.820785 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.820640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrl9\" (UniqueName: \"kubernetes.io/projected/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-kube-api-access-4jrl9\") pod \"maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:18.820785 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.820679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:18.820890 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:58:18.820787    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-serving-cert: secret "maas-api-e2e-ipp-a-a3cf0e-serving-cert" not found
Jul 24 00:58:18.820890 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:58:18.820880    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-maas-api-tls podName:69ff3965-b294-4e4a-8a6d-e6bcf4ac6886 nodeName:}" failed. No retries permitted until 2026-07-24 00:58:19.320858902 +0000 UTC m=+2679.888675030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-maas-api-tls") pod "maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv" (UID: "69ff3965-b294-4e4a-8a6d-e6bcf4ac6886") : secret "maas-api-e2e-ipp-a-a3cf0e-serving-cert" not found
Jul 24 00:58:18.821326 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.821302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:18.837386 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:18.837359    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrl9\" (UniqueName: \"kubernetes.io/projected/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-kube-api-access-4jrl9\") pod \"maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:19.323974 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:19.323935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:19.326371 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:19.326351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:19.520004 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:19.519962    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:19.663750 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:19.663715    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"]
Jul 24 00:58:19.666691 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:58:19.666654    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ff3965_b294_4e4a_8a6d_e6bcf4ac6886.slice/crio-3f6390bf5943142b542fa9fd4ab48e7dac5b56a6933df2e0e74fed92983bd8c1 WatchSource:0}: Error finding container 3f6390bf5943142b542fa9fd4ab48e7dac5b56a6933df2e0e74fed92983bd8c1: Status 404 returned error can't find the container with id 3f6390bf5943142b542fa9fd4ab48e7dac5b56a6933df2e0e74fed92983bd8c1
Jul 24 00:58:20.388681 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:20.388584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv" event={"ID":"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886","Type":"ContainerStarted","Data":"b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68"}
Jul 24 00:58:20.388681 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:20.388636    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv" event={"ID":"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886","Type":"ContainerStarted","Data":"3f6390bf5943142b542fa9fd4ab48e7dac5b56a6933df2e0e74fed92983bd8c1"}
Jul 24 00:58:20.388921 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:20.388716    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:20.410279 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:20.410224    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv" podStartSLOduration=1.976204512 podStartE2EDuration="2.410206706s" totalImagesPullingTime="434.002194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:58:20.407932529 +0000 UTC m=+2680.975748679" watchObservedRunningTime="2026-07-24 00:58:20.410206706 +0000 UTC m=+2680.978022857"
Jul 24 00:58:26.400195 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:26.400162    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 00:58:47.685035 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.684995    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"]
Jul 24 00:58:47.691029 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.690994    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:47.693845 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.693797    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-86b8da-serving-cert\""
Jul 24 00:58:47.700813 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.700783    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"]
Jul 24 00:58:47.782097 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.782064    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:47.782097 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.782106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:47.782368 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.782128    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zld4g\" (UniqueName: \"kubernetes.io/projected/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-kube-api-access-zld4g\") pod \"maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:47.883292 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.883255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:47.883469 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.883306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:47.883469 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.883329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zld4g\" (UniqueName: \"kubernetes.io/projected/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-kube-api-access-zld4g\") pod \"maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:47.883469 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:58:47.883429    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-serving-cert: secret "maas-api-e2e-ipp-b-86b8da-serving-cert" not found
Jul 24 00:58:47.883590 ip-10-0-135-176 kubenswrapper[2560]: E0724 00:58:47.883518    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-maas-api-tls podName:9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b nodeName:}" failed. No retries permitted until 2026-07-24 00:58:48.383491731 +0000 UTC m=+2708.951307876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-maas-api-tls") pod "maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs" (UID: "9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b") : secret "maas-api-e2e-ipp-b-86b8da-serving-cert" not found
Jul 24 00:58:47.883995 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.883979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:47.894590 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:47.894554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zld4g\" (UniqueName: \"kubernetes.io/projected/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-kube-api-access-zld4g\") pod \"maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:48.388979 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.388942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:48.391501 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.391481    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:48.447613 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.447577    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"]
Jul 24 00:58:48.451328 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.451310    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 00:58:48.453825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.453798    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-86b8da\""
Jul 24 00:58:48.453825 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.453800    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-86b8da-dockercfg-q779t\""
Jul 24 00:58:48.459351 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.459325    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"]
Jul 24 00:58:48.590967 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.590931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac9eb27-e91d-4b67-aa82-c77d3d175780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v\" (UID: \"1ac9eb27-e91d-4b67-aa82-c77d3d175780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 00:58:48.591147 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.591027    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwnb\" (UniqueName: \"kubernetes.io/projected/1ac9eb27-e91d-4b67-aa82-c77d3d175780-kube-api-access-9lwnb\") pod \"payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v\" (UID: \"1ac9eb27-e91d-4b67-aa82-c77d3d175780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 00:58:48.606029 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.606001    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:48.692392 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.692361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwnb\" (UniqueName: \"kubernetes.io/projected/1ac9eb27-e91d-4b67-aa82-c77d3d175780-kube-api-access-9lwnb\") pod \"payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v\" (UID: \"1ac9eb27-e91d-4b67-aa82-c77d3d175780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 00:58:48.692731 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.692427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac9eb27-e91d-4b67-aa82-c77d3d175780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v\" (UID: \"1ac9eb27-e91d-4b67-aa82-c77d3d175780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 00:58:48.693448 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.693419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac9eb27-e91d-4b67-aa82-c77d3d175780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v\" (UID: \"1ac9eb27-e91d-4b67-aa82-c77d3d175780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 00:58:48.701783 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.701755    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwnb\" (UniqueName: \"kubernetes.io/projected/1ac9eb27-e91d-4b67-aa82-c77d3d175780-kube-api-access-9lwnb\") pod \"payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v\" (UID: \"1ac9eb27-e91d-4b67-aa82-c77d3d175780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 00:58:48.740513 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.740485    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"]
Jul 24 00:58:48.742796 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:58:48.742769    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b24ae48_b86a_4b5c_80cb_9c8c6220cf5b.slice/crio-80a70fd6af11b1a5c6bae29127a3a598ad009091baa8d57bdb61ca58d1caf40d WatchSource:0}: Error finding container 80a70fd6af11b1a5c6bae29127a3a598ad009091baa8d57bdb61ca58d1caf40d: Status 404 returned error can't find the container with id 80a70fd6af11b1a5c6bae29127a3a598ad009091baa8d57bdb61ca58d1caf40d
Jul 24 00:58:48.763767 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.763737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 00:58:48.901886 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:48.901856    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"]
Jul 24 00:58:48.906269 ip-10-0-135-176 kubenswrapper[2560]: W0724 00:58:48.906220    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac9eb27_e91d_4b67_aa82_c77d3d175780.slice/crio-479cd808d1c2dd8064fcf559b4043447b7d29f1a514a7807762689273c178a11 WatchSource:0}: Error finding container 479cd808d1c2dd8064fcf559b4043447b7d29f1a514a7807762689273c178a11: Status 404 returned error can't find the container with id 479cd808d1c2dd8064fcf559b4043447b7d29f1a514a7807762689273c178a11
Jul 24 00:58:49.530779 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:49.530746    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs" event={"ID":"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b","Type":"ContainerStarted","Data":"d6c752f6d551388bc9674ab1c2e3fb424c19f83894470dc09b13fa4f1be103a2"}
Jul 24 00:58:49.530999 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:49.530795    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs" event={"ID":"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b","Type":"ContainerStarted","Data":"80a70fd6af11b1a5c6bae29127a3a598ad009091baa8d57bdb61ca58d1caf40d"}
Jul 24 00:58:49.530999 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:49.530878    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:58:49.532701 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:49.532673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v" event={"ID":"1ac9eb27-e91d-4b67-aa82-c77d3d175780","Type":"ContainerStarted","Data":"cabfcfa86f7eb4bb7013e1a1905b81f9703ceef378cfd55b858788674d997863"}
Jul 24 00:58:49.532821 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:49.532720    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v" event={"ID":"1ac9eb27-e91d-4b67-aa82-c77d3d175780","Type":"ContainerStarted","Data":"479cd808d1c2dd8064fcf559b4043447b7d29f1a514a7807762689273c178a11"}
Jul 24 00:58:49.532821 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:49.532754    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 00:58:49.549486 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:49.549431    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs" podStartSLOduration=2.227227916 podStartE2EDuration="2.549412931s" totalImagesPullingTime="322.185015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:58:49.548904197 +0000 UTC m=+2710.116720348" watchObservedRunningTime="2026-07-24 00:58:49.549412931 +0000 UTC m=+2710.117229081"
Jul 24 00:58:49.565995 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:49.565942    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v" podStartSLOduration=1.565926683 podStartE2EDuration="1.565926683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:58:49.564887575 +0000 UTC m=+2710.132703727" watchObservedRunningTime="2026-07-24 00:58:49.565926683 +0000 UTC m=+2710.133742832"
Jul 24 00:58:55.543579 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:58:55.543543    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 00:59:00.539917 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:59:00.539878    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 00:59:09.415235 ip-10-0-135-176 kubenswrapper[2560]: I0724 00:59:09.415194    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-nw6g8_695ae2f8-1dcd-4555-b5b1-f683cacb89e4/payload-processing/0.log"
Jul 24 01:00:00.134046 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.134004    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms"]
Jul 24 01:00:00.137396 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.137373    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms"
Jul 24 01:00:00.139637 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.139609    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\""
Jul 24 01:00:00.140786 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.140765    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2"]
Jul 24 01:00:00.144271 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.144245    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557"]
Jul 24 01:00:00.144498 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.144476    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2"
Jul 24 01:00:00.148169 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.148146    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms"]
Jul 24 01:00:00.148289 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.148240    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557"
Jul 24 01:00:00.151278 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.151254    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms"
Jul 24 01:00:00.162039 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.161972    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2"]
Jul 24 01:00:00.165228 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.165202    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557"]
Jul 24 01:00:00.165416 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.165393    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2"
Jul 24 01:00:00.211814 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.211775    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557"
Jul 24 01:00:00.326927 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.326897    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms"]
Jul 24 01:00:00.329803 ip-10-0-135-176 kubenswrapper[2560]: W0724 01:00:00.329766    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9f9a8e_ffa7_4aa8_adc5_0e1397421db3.slice/crio-290499345a94366ab34da8d5e8f0699aa843530debc6cf219235e1f9d1659b6b WatchSource:0}: Error finding container 290499345a94366ab34da8d5e8f0699aa843530debc6cf219235e1f9d1659b6b: Status 404 returned error can't find the container with id 290499345a94366ab34da8d5e8f0699aa843530debc6cf219235e1f9d1659b6b
Jul 24 01:00:00.391243 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.391173    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557"]
Jul 24 01:00:00.394447 ip-10-0-135-176 kubenswrapper[2560]: W0724 01:00:00.394414    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47db328d_d561_47d7_8c25_a3a7bc9c6c06.slice/crio-c296e2d097efbf001298bc5d4b31beb8a3ded9c035a4fe6d49c49e12a16aea3f WatchSource:0}: Error finding container c296e2d097efbf001298bc5d4b31beb8a3ded9c035a4fe6d49c49e12a16aea3f: Status 404 returned error can't find the container with id c296e2d097efbf001298bc5d4b31beb8a3ded9c035a4fe6d49c49e12a16aea3f
Jul 24 01:00:00.562331 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.562306    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2"]
Jul 24 01:00:00.564286 ip-10-0-135-176 kubenswrapper[2560]: W0724 01:00:00.564263    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb895a037_979a_4336_9891_99b3ef6de8fb.slice/crio-3e02bc268aea43858b7dee902e9f6573db0f08b697c31465484ce5361c75703a WatchSource:0}: Error finding container 3e02bc268aea43858b7dee902e9f6573db0f08b697c31465484ce5361c75703a: Status 404 returned error can't find the container with id 3e02bc268aea43858b7dee902e9f6573db0f08b697c31465484ce5361c75703a
Jul 24 01:00:00.876433 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.876394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557" event={"ID":"47db328d-d561-47d7-8c25-a3a7bc9c6c06","Type":"ContainerStarted","Data":"beccd89f1aa670d3ce5eba847dc42f283161b5e8212047f67165ac40abf4bd81"}
Jul 24 01:00:00.876631 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.876457    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557" event={"ID":"47db328d-d561-47d7-8c25-a3a7bc9c6c06","Type":"ContainerStarted","Data":"c296e2d097efbf001298bc5d4b31beb8a3ded9c035a4fe6d49c49e12a16aea3f"}
Jul 24 01:00:00.877924 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.877897    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms" event={"ID":"ca9f9a8e-ffa7-4aa8-adc5-0e1397421db3","Type":"ContainerStarted","Data":"61837f5193566b2f0131a6ff1a23e3cd875e6b44b04b84710e98f8adab32d95e"}
Jul 24 01:00:00.878024 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.877939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms" event={"ID":"ca9f9a8e-ffa7-4aa8-adc5-0e1397421db3","Type":"ContainerStarted","Data":"290499345a94366ab34da8d5e8f0699aa843530debc6cf219235e1f9d1659b6b"}
Jul 24 01:00:00.879392 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.879371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2" event={"ID":"b895a037-979a-4336-9891-99b3ef6de8fb","Type":"ContainerStarted","Data":"c39344ddda27383e9e7250557576f84aa83c9e4a1525c4c4c478e4981879374b"}
Jul 24 01:00:00.879478 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.879408    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2" event={"ID":"b895a037-979a-4336-9891-99b3ef6de8fb","Type":"ContainerStarted","Data":"3e02bc268aea43858b7dee902e9f6573db0f08b697c31465484ce5361c75703a"}
Jul 24 01:00:00.892590 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.892542    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557" podStartSLOduration=0.89252864 podStartE2EDuration="892.52864ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:00:00.890579028 +0000 UTC m=+2781.458395178" watchObservedRunningTime="2026-07-24 01:00:00.89252864 +0000 UTC m=+2781.460344846"
Jul 24 01:00:00.906123 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.906021    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms" podStartSLOduration=0.90600632 podStartE2EDuration="906.00632ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:00:00.904025367 +0000 UTC m=+2781.471841541" watchObservedRunningTime="2026-07-24 01:00:00.90600632 +0000 UTC m=+2781.473822470"
Jul 24 01:00:00.919513 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:00.919465    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2" podStartSLOduration=0.919447779 podStartE2EDuration="919.447779ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:00:00.918393699 +0000 UTC m=+2781.486209846" watchObservedRunningTime="2026-07-24 01:00:00.919447779 +0000 UTC m=+2781.487263929"
Jul 24 01:00:01.886435 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:01.886403    2560 generic.go:356] "Generic (PLEG): container finished" podID="47db328d-d561-47d7-8c25-a3a7bc9c6c06" containerID="beccd89f1aa670d3ce5eba847dc42f283161b5e8212047f67165ac40abf4bd81" exitCode=0
Jul 24 01:00:01.886903 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:01.886488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557" event={"ID":"47db328d-d561-47d7-8c25-a3a7bc9c6c06","Type":"ContainerDied","Data":"beccd89f1aa670d3ce5eba847dc42f283161b5e8212047f67165ac40abf4bd81"}
Jul 24 01:00:01.888276 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:01.888251    2560 generic.go:356] "Generic (PLEG): container finished" podID="ca9f9a8e-ffa7-4aa8-adc5-0e1397421db3" containerID="61837f5193566b2f0131a6ff1a23e3cd875e6b44b04b84710e98f8adab32d95e" exitCode=0
Jul 24 01:00:01.888390 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:01.888327    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms" event={"ID":"ca9f9a8e-ffa7-4aa8-adc5-0e1397421db3","Type":"ContainerDied","Data":"61837f5193566b2f0131a6ff1a23e3cd875e6b44b04b84710e98f8adab32d95e"}
Jul 24 01:00:01.889710 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:01.889687    2560 generic.go:356] "Generic (PLEG): container finished" podID="b895a037-979a-4336-9891-99b3ef6de8fb" containerID="c39344ddda27383e9e7250557576f84aa83c9e4a1525c4c4c478e4981879374b" exitCode=0
Jul 24 01:00:01.889792 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:01.889758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2" event={"ID":"b895a037-979a-4336-9891-99b3ef6de8fb","Type":"ContainerDied","Data":"c39344ddda27383e9e7250557576f84aa83c9e4a1525c4c4c478e4981879374b"}
Jul 24 01:00:03.041175 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.041142    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2"
Jul 24 01:00:03.090338 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.090307    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms"
Jul 24 01:00:03.093482 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.093459    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557"
Jul 24 01:00:03.901252 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.901222    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms"
Jul 24 01:00:03.901252 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.901233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms" event={"ID":"ca9f9a8e-ffa7-4aa8-adc5-0e1397421db3","Type":"ContainerDied","Data":"290499345a94366ab34da8d5e8f0699aa843530debc6cf219235e1f9d1659b6b"}
Jul 24 01:00:03.901698 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.901285    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="290499345a94366ab34da8d5e8f0699aa843530debc6cf219235e1f9d1659b6b"
Jul 24 01:00:03.902608 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.902582    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2"
Jul 24 01:00:03.902743 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.902612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2" event={"ID":"b895a037-979a-4336-9891-99b3ef6de8fb","Type":"ContainerDied","Data":"3e02bc268aea43858b7dee902e9f6573db0f08b697c31465484ce5361c75703a"}
Jul 24 01:00:03.902743 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.902661    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e02bc268aea43858b7dee902e9f6573db0f08b697c31465484ce5361c75703a"
Jul 24 01:00:03.903888 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.903854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557" event={"ID":"47db328d-d561-47d7-8c25-a3a7bc9c6c06","Type":"ContainerDied","Data":"c296e2d097efbf001298bc5d4b31beb8a3ded9c035a4fe6d49c49e12a16aea3f"}
Jul 24 01:00:03.903888 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.903876    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557"
Jul 24 01:00:03.904047 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:03.903899    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c296e2d097efbf001298bc5d4b31beb8a3ded9c035a4fe6d49c49e12a16aea3f"
Jul 24 01:00:04.741640 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:04.741608    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-nw6g8_695ae2f8-1dcd-4555-b5b1-f683cacb89e4/payload-processing/0.log"
Jul 24 01:00:08.333126 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.333091    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"]
Jul 24 01:00:08.333720 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.333703    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47db328d-d561-47d7-8c25-a3a7bc9c6c06" containerName="cleanup"
Jul 24 01:00:08.333768 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.333722    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="47db328d-d561-47d7-8c25-a3a7bc9c6c06" containerName="cleanup"
Jul 24 01:00:08.333768 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.333748    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9f9a8e-ffa7-4aa8-adc5-0e1397421db3" containerName="cleanup"
Jul 24 01:00:08.333768 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.333756    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9f9a8e-ffa7-4aa8-adc5-0e1397421db3" containerName="cleanup"
Jul 24 01:00:08.333768 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.333765    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b895a037-979a-4336-9891-99b3ef6de8fb" containerName="cleanup"
Jul 24 01:00:08.333919 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.333775    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b895a037-979a-4336-9891-99b3ef6de8fb" containerName="cleanup"
Jul 24 01:00:08.333919 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.333909    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="47db328d-d561-47d7-8c25-a3a7bc9c6c06" containerName="cleanup"
Jul 24 01:00:08.333980 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.333924    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b895a037-979a-4336-9891-99b3ef6de8fb" containerName="cleanup"
Jul 24 01:00:08.333980 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.333937    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9f9a8e-ffa7-4aa8-adc5-0e1397421db3" containerName="cleanup"
Jul 24 01:00:08.339415 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.339387    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:08.342246 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.342220    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-e040d3-serving-cert\""
Jul 24 01:00:08.358629 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.358598    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"]
Jul 24 01:00:08.409660 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.409628    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61583a24-39d3-4935-9984-bdb6815fc070-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:08.409884 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.409704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qms\" (UniqueName: \"kubernetes.io/projected/61583a24-39d3-4935-9984-bdb6815fc070-kube-api-access-m5qms\") pod \"maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:08.409884 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.409736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61583a24-39d3-4935-9984-bdb6815fc070-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:08.510634 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.510595    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qms\" (UniqueName: \"kubernetes.io/projected/61583a24-39d3-4935-9984-bdb6815fc070-kube-api-access-m5qms\") pod \"maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:08.510888 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.510662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61583a24-39d3-4935-9984-bdb6815fc070-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:08.510888 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.510710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61583a24-39d3-4935-9984-bdb6815fc070-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:08.511061 ip-10-0-135-176 kubenswrapper[2560]: E0724 01:00:08.510906    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-serving-cert: secret "maas-api-e2e-ipp-cleanup-e040d3-serving-cert" not found
Jul 24 01:00:08.511061 ip-10-0-135-176 kubenswrapper[2560]: E0724 01:00:08.510984    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61583a24-39d3-4935-9984-bdb6815fc070-maas-api-tls podName:61583a24-39d3-4935-9984-bdb6815fc070 nodeName:}" failed. No retries permitted until 2026-07-24 01:00:09.010961705 +0000 UTC m=+2789.578777848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61583a24-39d3-4935-9984-bdb6815fc070-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj" (UID: "61583a24-39d3-4935-9984-bdb6815fc070") : secret "maas-api-e2e-ipp-cleanup-e040d3-serving-cert" not found
Jul 24 01:00:08.511316 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.511296    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61583a24-39d3-4935-9984-bdb6815fc070-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:08.520344 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:08.520316    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qms\" (UniqueName: \"kubernetes.io/projected/61583a24-39d3-4935-9984-bdb6815fc070-kube-api-access-m5qms\") pod \"maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:09.015153 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.015107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61583a24-39d3-4935-9984-bdb6815fc070-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:09.017740 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.017712    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61583a24-39d3-4935-9984-bdb6815fc070-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:09.142662 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.142625    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"]
Jul 24 01:00:09.146387 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.146367    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:09.148746 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.148722    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-e040d3\""
Jul 24 01:00:09.148880 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.148854    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-e040d3-dockercfg-dvmdx\""
Jul 24 01:00:09.156166 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.156141    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"]
Jul 24 01:00:09.173935 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.173907    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"]
Jul 24 01:00:09.177584 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.177565    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:09.188381 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.188357    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"]
Jul 24 01:00:09.216542 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.216509    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180c0fd-e2c6-4422-838b-bcf75121496e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s\" (UID: \"0180c0fd-e2c6-4422-838b-bcf75121496e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:09.216682 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.216572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p7jl\" (UniqueName: \"kubernetes.io/projected/0180c0fd-e2c6-4422-838b-bcf75121496e-kube-api-access-7p7jl\") pod \"payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s\" (UID: \"0180c0fd-e2c6-4422-838b-bcf75121496e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:09.251502 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.251469    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:09.317316 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.317115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180c0fd-e2c6-4422-838b-bcf75121496e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s\" (UID: \"0180c0fd-e2c6-4422-838b-bcf75121496e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:09.317316 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.317188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5cd9\" (UniqueName: \"kubernetes.io/projected/7d04217b-9040-420e-a51d-05a583f6ab7f-kube-api-access-z5cd9\") pod \"payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz\" (UID: \"7d04217b-9040-420e-a51d-05a583f6ab7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:09.317316 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.317238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p7jl\" (UniqueName: \"kubernetes.io/projected/0180c0fd-e2c6-4422-838b-bcf75121496e-kube-api-access-7p7jl\") pod \"payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s\" (UID: \"0180c0fd-e2c6-4422-838b-bcf75121496e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:09.317316 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.317319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04217b-9040-420e-a51d-05a583f6ab7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz\" (UID: \"7d04217b-9040-420e-a51d-05a583f6ab7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:09.318864 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.318036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180c0fd-e2c6-4422-838b-bcf75121496e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s\" (UID: \"0180c0fd-e2c6-4422-838b-bcf75121496e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:09.329107 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.329076    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p7jl\" (UniqueName: \"kubernetes.io/projected/0180c0fd-e2c6-4422-838b-bcf75121496e-kube-api-access-7p7jl\") pod \"payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s\" (UID: \"0180c0fd-e2c6-4422-838b-bcf75121496e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:09.399977 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.399948    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"]
Jul 24 01:00:09.401316 ip-10-0-135-176 kubenswrapper[2560]: W0724 01:00:09.401281    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61583a24_39d3_4935_9984_bdb6815fc070.slice/crio-1b95392ba0ff2af202ca3c6740fad4a6e4be03f3113abd463e0e3cd9236bc02a WatchSource:0}: Error finding container 1b95392ba0ff2af202ca3c6740fad4a6e4be03f3113abd463e0e3cd9236bc02a: Status 404 returned error can't find the container with id 1b95392ba0ff2af202ca3c6740fad4a6e4be03f3113abd463e0e3cd9236bc02a
Jul 24 01:00:09.419255 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.419217    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5cd9\" (UniqueName: \"kubernetes.io/projected/7d04217b-9040-420e-a51d-05a583f6ab7f-kube-api-access-z5cd9\") pod \"payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz\" (UID: \"7d04217b-9040-420e-a51d-05a583f6ab7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:09.419405 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.419290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04217b-9040-420e-a51d-05a583f6ab7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz\" (UID: \"7d04217b-9040-420e-a51d-05a583f6ab7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:09.420157 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.420131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04217b-9040-420e-a51d-05a583f6ab7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz\" (UID: \"7d04217b-9040-420e-a51d-05a583f6ab7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:09.426678 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.426654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5cd9\" (UniqueName: \"kubernetes.io/projected/7d04217b-9040-420e-a51d-05a583f6ab7f-kube-api-access-z5cd9\") pod \"payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz\" (UID: \"7d04217b-9040-420e-a51d-05a583f6ab7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:09.459361 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.459323    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:09.490734 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.490705    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:09.645284 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.645230    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"]
Jul 24 01:00:09.648948 ip-10-0-135-176 kubenswrapper[2560]: W0724 01:00:09.648886    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0180c0fd_e2c6_4422_838b_bcf75121496e.slice/crio-28890f2fd090db3dd1970ec2b3fa7eed0dd0bd0047a95d63bac2b498cd39122f WatchSource:0}: Error finding container 28890f2fd090db3dd1970ec2b3fa7eed0dd0bd0047a95d63bac2b498cd39122f: Status 404 returned error can't find the container with id 28890f2fd090db3dd1970ec2b3fa7eed0dd0bd0047a95d63bac2b498cd39122f
Jul 24 01:00:09.680105 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.680075    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"]
Jul 24 01:00:09.693051 ip-10-0-135-176 kubenswrapper[2560]: W0724 01:00:09.693020    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d04217b_9040_420e_a51d_05a583f6ab7f.slice/crio-381858638a3d75478d1893769f5b82da1e0997526e84a49ccdbae98a65e2d355 WatchSource:0}: Error finding container 381858638a3d75478d1893769f5b82da1e0997526e84a49ccdbae98a65e2d355: Status 404 returned error can't find the container with id 381858638a3d75478d1893769f5b82da1e0997526e84a49ccdbae98a65e2d355
Jul 24 01:00:09.934003 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.933923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz" event={"ID":"7d04217b-9040-420e-a51d-05a583f6ab7f","Type":"ContainerStarted","Data":"5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9"}
Jul 24 01:00:09.934003 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.933989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz" event={"ID":"7d04217b-9040-420e-a51d-05a583f6ab7f","Type":"ContainerStarted","Data":"381858638a3d75478d1893769f5b82da1e0997526e84a49ccdbae98a65e2d355"}
Jul 24 01:00:09.934232 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.934108    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:09.935608 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.935580    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s" event={"ID":"0180c0fd-e2c6-4422-838b-bcf75121496e","Type":"ContainerStarted","Data":"2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e"}
Jul 24 01:00:09.935760 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.935630    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s" event={"ID":"0180c0fd-e2c6-4422-838b-bcf75121496e","Type":"ContainerStarted","Data":"28890f2fd090db3dd1970ec2b3fa7eed0dd0bd0047a95d63bac2b498cd39122f"}
Jul 24 01:00:09.935760 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.935690    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:09.937108 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.937080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj" event={"ID":"61583a24-39d3-4935-9984-bdb6815fc070","Type":"ContainerStarted","Data":"1b95392ba0ff2af202ca3c6740fad4a6e4be03f3113abd463e0e3cd9236bc02a"}
Jul 24 01:00:09.971263 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:09.971129    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz" podStartSLOduration=0.971110391 podStartE2EDuration="971.110391ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:00:09.969063991 +0000 UTC m=+2790.536880147" watchObservedRunningTime="2026-07-24 01:00:09.971110391 +0000 UTC m=+2790.538926540"
Jul 24 01:00:10.014306 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:10.014257    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s" podStartSLOduration=1.01424238 podStartE2EDuration="1.01424238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:00:10.012409383 +0000 UTC m=+2790.580225533" watchObservedRunningTime="2026-07-24 01:00:10.01424238 +0000 UTC m=+2790.582058553"
Jul 24 01:00:10.944980 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:10.944944    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj" event={"ID":"61583a24-39d3-4935-9984-bdb6815fc070","Type":"ContainerStarted","Data":"e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921"}
Jul 24 01:00:10.964752 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:10.964699    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj" podStartSLOduration=2.414684167 podStartE2EDuration="2.964684654s" totalImagesPullingTime="550.000487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:00:10.961030324 +0000 UTC m=+2791.528846467" watchObservedRunningTime="2026-07-24 01:00:10.964684654 +0000 UTC m=+2791.532500805"
Jul 24 01:00:11.951083 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:11.951053    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:17.968096 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:17.968065    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:20.946782 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:20.946747    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:20.947239 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:20.946824    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:34.065185 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.065149    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"]
Jul 24 01:00:34.065669 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.065425    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj" podUID="61583a24-39d3-4935-9984-bdb6815fc070" containerName="maas-api" containerID="cri-o://e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921" gracePeriod=30
Jul 24 01:00:34.126909 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.126871    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"]
Jul 24 01:00:34.127177 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.127149    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz" podUID="7d04217b-9040-420e-a51d-05a583f6ab7f" containerName="payload-processing" containerID="cri-o://5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9" gracePeriod=30
Jul 24 01:00:34.135710 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.135107    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"]
Jul 24 01:00:34.135710 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.135339    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s" podUID="0180c0fd-e2c6-4422-838b-bcf75121496e" containerName="payload-pre-processing" containerID="cri-o://2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e" gracePeriod=30
Jul 24 01:00:34.400990 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.400958    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:34.435336 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.435296    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:34.445240 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.445214    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:34.447407 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.447383    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61583a24-39d3-4935-9984-bdb6815fc070-openshift-service-ca\") pod \"61583a24-39d3-4935-9984-bdb6815fc070\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") "
Jul 24 01:00:34.447502 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.447467    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61583a24-39d3-4935-9984-bdb6815fc070-maas-api-tls\") pod \"61583a24-39d3-4935-9984-bdb6815fc070\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") "
Jul 24 01:00:34.447502 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.447496    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p7jl\" (UniqueName: \"kubernetes.io/projected/0180c0fd-e2c6-4422-838b-bcf75121496e-kube-api-access-7p7jl\") pod \"0180c0fd-e2c6-4422-838b-bcf75121496e\" (UID: \"0180c0fd-e2c6-4422-838b-bcf75121496e\") "
Jul 24 01:00:34.447626 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.447521    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04217b-9040-420e-a51d-05a583f6ab7f-plugins-config-volume\") pod \"7d04217b-9040-420e-a51d-05a583f6ab7f\" (UID: \"7d04217b-9040-420e-a51d-05a583f6ab7f\") "
Jul 24 01:00:34.447689 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.447622    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5qms\" (UniqueName: \"kubernetes.io/projected/61583a24-39d3-4935-9984-bdb6815fc070-kube-api-access-m5qms\") pod \"61583a24-39d3-4935-9984-bdb6815fc070\" (UID: \"61583a24-39d3-4935-9984-bdb6815fc070\") "
Jul 24 01:00:34.447742 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.447686    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5cd9\" (UniqueName: \"kubernetes.io/projected/7d04217b-9040-420e-a51d-05a583f6ab7f-kube-api-access-z5cd9\") pod \"7d04217b-9040-420e-a51d-05a583f6ab7f\" (UID: \"7d04217b-9040-420e-a51d-05a583f6ab7f\") "
Jul 24 01:00:34.447742 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.447714    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180c0fd-e2c6-4422-838b-bcf75121496e-plugins-config-volume\") pod \"0180c0fd-e2c6-4422-838b-bcf75121496e\" (UID: \"0180c0fd-e2c6-4422-838b-bcf75121496e\") "
Jul 24 01:00:34.447862 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.447790    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61583a24-39d3-4935-9984-bdb6815fc070-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "61583a24-39d3-4935-9984-bdb6815fc070" (UID: "61583a24-39d3-4935-9984-bdb6815fc070"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:00:34.447978 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.447933    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d04217b-9040-420e-a51d-05a583f6ab7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d04217b-9040-420e-a51d-05a583f6ab7f" (UID: "7d04217b-9040-420e-a51d-05a583f6ab7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:00:34.448217 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.448044    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04217b-9040-420e-a51d-05a583f6ab7f-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:00:34.448217 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.448070    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61583a24-39d3-4935-9984-bdb6815fc070-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:00:34.448217 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.448158    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0180c0fd-e2c6-4422-838b-bcf75121496e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0180c0fd-e2c6-4422-838b-bcf75121496e" (UID: "0180c0fd-e2c6-4422-838b-bcf75121496e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:00:34.449712 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.449686    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0180c0fd-e2c6-4422-838b-bcf75121496e-kube-api-access-7p7jl" (OuterVolumeSpecName: "kube-api-access-7p7jl") pod "0180c0fd-e2c6-4422-838b-bcf75121496e" (UID: "0180c0fd-e2c6-4422-838b-bcf75121496e"). InnerVolumeSpecName "kube-api-access-7p7jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:00:34.450024 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.449940    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61583a24-39d3-4935-9984-bdb6815fc070-kube-api-access-m5qms" (OuterVolumeSpecName: "kube-api-access-m5qms") pod "61583a24-39d3-4935-9984-bdb6815fc070" (UID: "61583a24-39d3-4935-9984-bdb6815fc070"). InnerVolumeSpecName "kube-api-access-m5qms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:00:34.450412 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.450389    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d04217b-9040-420e-a51d-05a583f6ab7f-kube-api-access-z5cd9" (OuterVolumeSpecName: "kube-api-access-z5cd9") pod "7d04217b-9040-420e-a51d-05a583f6ab7f" (UID: "7d04217b-9040-420e-a51d-05a583f6ab7f"). InnerVolumeSpecName "kube-api-access-z5cd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:00:34.450481 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.450408    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61583a24-39d3-4935-9984-bdb6815fc070-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "61583a24-39d3-4935-9984-bdb6815fc070" (UID: "61583a24-39d3-4935-9984-bdb6815fc070"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:00:34.548974 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.548938    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61583a24-39d3-4935-9984-bdb6815fc070-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:00:34.548974 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.548972    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p7jl\" (UniqueName: \"kubernetes.io/projected/0180c0fd-e2c6-4422-838b-bcf75121496e-kube-api-access-7p7jl\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:00:34.549176 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.548984    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qms\" (UniqueName: \"kubernetes.io/projected/61583a24-39d3-4935-9984-bdb6815fc070-kube-api-access-m5qms\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:00:34.549176 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.548996    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5cd9\" (UniqueName: \"kubernetes.io/projected/7d04217b-9040-420e-a51d-05a583f6ab7f-kube-api-access-z5cd9\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:00:34.549176 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:34.549009    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180c0fd-e2c6-4422-838b-bcf75121496e-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:00:35.071797 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.071759    2560 generic.go:356] "Generic (PLEG): container finished" podID="61583a24-39d3-4935-9984-bdb6815fc070" containerID="e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921" exitCode=0
Jul 24 01:00:35.072308 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.071803    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"
Jul 24 01:00:35.072308 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.071852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj" event={"ID":"61583a24-39d3-4935-9984-bdb6815fc070","Type":"ContainerDied","Data":"e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921"}
Jul 24 01:00:35.072308 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.071908    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj" event={"ID":"61583a24-39d3-4935-9984-bdb6815fc070","Type":"ContainerDied","Data":"1b95392ba0ff2af202ca3c6740fad4a6e4be03f3113abd463e0e3cd9236bc02a"}
Jul 24 01:00:35.072308 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.071936    2560 scope.go:117] "RemoveContainer" containerID="e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921"
Jul 24 01:00:35.073544 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.073521    2560 generic.go:356] "Generic (PLEG): container finished" podID="7d04217b-9040-420e-a51d-05a583f6ab7f" containerID="5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9" exitCode=0
Jul 24 01:00:35.073643 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.073551    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"
Jul 24 01:00:35.073643 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.073604    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz" event={"ID":"7d04217b-9040-420e-a51d-05a583f6ab7f","Type":"ContainerDied","Data":"5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9"}
Jul 24 01:00:35.073748 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.073651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz" event={"ID":"7d04217b-9040-420e-a51d-05a583f6ab7f","Type":"ContainerDied","Data":"381858638a3d75478d1893769f5b82da1e0997526e84a49ccdbae98a65e2d355"}
Jul 24 01:00:35.075302 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.075279    2560 generic.go:356] "Generic (PLEG): container finished" podID="0180c0fd-e2c6-4422-838b-bcf75121496e" containerID="2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e" exitCode=0
Jul 24 01:00:35.075425 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.075313    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s" event={"ID":"0180c0fd-e2c6-4422-838b-bcf75121496e","Type":"ContainerDied","Data":"2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e"}
Jul 24 01:00:35.075425 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.075354    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s" event={"ID":"0180c0fd-e2c6-4422-838b-bcf75121496e","Type":"ContainerDied","Data":"28890f2fd090db3dd1970ec2b3fa7eed0dd0bd0047a95d63bac2b498cd39122f"}
Jul 24 01:00:35.075425 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.075419    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"
Jul 24 01:00:35.095349 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.095258    2560 scope.go:117] "RemoveContainer" containerID="e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921"
Jul 24 01:00:35.095634 ip-10-0-135-176 kubenswrapper[2560]: E0724 01:00:35.095606    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921\": container with ID starting with e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921 not found: ID does not exist" containerID="e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921"
Jul 24 01:00:35.095731 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.095643    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921"} err="failed to get container status \"e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921\": rpc error: code = NotFound desc = could not find container \"e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921\": container with ID starting with e7ad3c7a99f838a16dac84e0e22525c1a09c4a1b989c6d860037860a15b93921 not found: ID does not exist"
Jul 24 01:00:35.095731 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.095660    2560 scope.go:117] "RemoveContainer" containerID="5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9"
Jul 24 01:00:35.110158 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.110122    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"]
Jul 24 01:00:35.111399 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.111375    2560 scope.go:117] "RemoveContainer" containerID="5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9"
Jul 24 01:00:35.111734 ip-10-0-135-176 kubenswrapper[2560]: E0724 01:00:35.111712    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9\": container with ID starting with 5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9 not found: ID does not exist" containerID="5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9"
Jul 24 01:00:35.111790 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.111744    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9"} err="failed to get container status \"5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9\": rpc error: code = NotFound desc = could not find container \"5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9\": container with ID starting with 5d8a8d888025be6f2246aeae009cc30e7b9868683f4d394f456f3d4eb28710e9 not found: ID does not exist"
Jul 24 01:00:35.111790 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.111764    2560 scope.go:117] "RemoveContainer" containerID="2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e"
Jul 24 01:00:35.115514 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.115480    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e040d3-7ffd69db74-lnhdj"]
Jul 24 01:00:35.120598 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.120575    2560 scope.go:117] "RemoveContainer" containerID="2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e"
Jul 24 01:00:35.120917 ip-10-0-135-176 kubenswrapper[2560]: E0724 01:00:35.120891    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e\": container with ID starting with 2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e not found: ID does not exist" containerID="2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e"
Jul 24 01:00:35.120986 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.120930    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e"} err="failed to get container status \"2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e\": rpc error: code = NotFound desc = could not find container \"2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e\": container with ID starting with 2526ee336739abefe9e3d0a557f1a6c7fa67f73f2ce58c5086eec913ec092f4e not found: ID does not exist"
Jul 24 01:00:35.124081 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.124058    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"]
Jul 24 01:00:35.127672 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.127648    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e040d3-6fcf58554d-7s27s"]
Jul 24 01:00:35.139402 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.139375    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"]
Jul 24 01:00:35.143713 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.143686    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e040d3-78b47894f6-zhnnz"]
Jul 24 01:00:35.968392 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.968358    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0180c0fd-e2c6-4422-838b-bcf75121496e" path="/var/lib/kubelet/pods/0180c0fd-e2c6-4422-838b-bcf75121496e/volumes"
Jul 24 01:00:35.968760 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.968747    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61583a24-39d3-4935-9984-bdb6815fc070" path="/var/lib/kubelet/pods/61583a24-39d3-4935-9984-bdb6815fc070/volumes"
Jul 24 01:00:35.969117 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:35.969104    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d04217b-9040-420e-a51d-05a583f6ab7f" path="/var/lib/kubelet/pods/7d04217b-9040-420e-a51d-05a583f6ab7f/volumes"
Jul 24 01:00:48.861312 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.861274    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"]
Jul 24 01:00:48.861710 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.861697    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0180c0fd-e2c6-4422-838b-bcf75121496e" containerName="payload-pre-processing"
Jul 24 01:00:48.861756 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.861712    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0180c0fd-e2c6-4422-838b-bcf75121496e" containerName="payload-pre-processing"
Jul 24 01:00:48.861756 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.861727    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d04217b-9040-420e-a51d-05a583f6ab7f" containerName="payload-processing"
Jul 24 01:00:48.861756 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.861732    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d04217b-9040-420e-a51d-05a583f6ab7f" containerName="payload-processing"
Jul 24 01:00:48.861756 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.861745    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61583a24-39d3-4935-9984-bdb6815fc070" containerName="maas-api"
Jul 24 01:00:48.861756 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.861751    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="61583a24-39d3-4935-9984-bdb6815fc070" containerName="maas-api"
Jul 24 01:00:48.861934 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.861809    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0180c0fd-e2c6-4422-838b-bcf75121496e" containerName="payload-pre-processing"
Jul 24 01:00:48.861934 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.861818    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="61583a24-39d3-4935-9984-bdb6815fc070" containerName="maas-api"
Jul 24 01:00:48.861934 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.861839    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d04217b-9040-420e-a51d-05a583f6ab7f" containerName="payload-processing"
Jul 24 01:00:48.866562 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.866543    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"
Jul 24 01:00:48.869282 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.869262    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\""
Jul 24 01:00:48.872047 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.872016    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"]
Jul 24 01:00:48.981425 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:48.981388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b454bc-04a9-4a82-ae03-94358fbe39c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh\" (UID: \"96b454bc-04a9-4a82-ae03-94358fbe39c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"
Jul 24 01:00:49.082777 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:49.082734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b454bc-04a9-4a82-ae03-94358fbe39c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh\" (UID: \"96b454bc-04a9-4a82-ae03-94358fbe39c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"
Jul 24 01:00:49.083434 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:49.083413    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b454bc-04a9-4a82-ae03-94358fbe39c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh\" (UID: \"96b454bc-04a9-4a82-ae03-94358fbe39c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"
Jul 24 01:00:49.180625 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:49.180540    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"
Jul 24 01:00:49.318786 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:49.318759    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"]
Jul 24 01:00:49.321212 ip-10-0-135-176 kubenswrapper[2560]: W0724 01:00:49.321172    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b454bc_04a9_4a82_ae03_94358fbe39c4.slice/crio-4d20805e871c5dcbbf6a349c87d7d4199a782b0b74f9d1981f5d2c0570f8f370 WatchSource:0}: Error finding container 4d20805e871c5dcbbf6a349c87d7d4199a782b0b74f9d1981f5d2c0570f8f370: Status 404 returned error can't find the container with id 4d20805e871c5dcbbf6a349c87d7d4199a782b0b74f9d1981f5d2c0570f8f370
Jul 24 01:00:50.156478 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:50.156436    2560 generic.go:356] "Generic (PLEG): container finished" podID="96b454bc-04a9-4a82-ae03-94358fbe39c4" containerID="8859671dc95622e60b14f48d5fc6bef2773b34ded07ff6f4a5250f068413ceef" exitCode=0
Jul 24 01:00:50.156856 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:50.156514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh" event={"ID":"96b454bc-04a9-4a82-ae03-94358fbe39c4","Type":"ContainerDied","Data":"8859671dc95622e60b14f48d5fc6bef2773b34ded07ff6f4a5250f068413ceef"}
Jul 24 01:00:50.156856 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:50.156572    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh" event={"ID":"96b454bc-04a9-4a82-ae03-94358fbe39c4","Type":"ContainerStarted","Data":"4d20805e871c5dcbbf6a349c87d7d4199a782b0b74f9d1981f5d2c0570f8f370"}
Jul 24 01:00:51.296158 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:51.296132    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"
Jul 24 01:00:51.403328 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:51.403293    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b454bc-04a9-4a82-ae03-94358fbe39c4-maas-api-service-ca\") pod \"96b454bc-04a9-4a82-ae03-94358fbe39c4\" (UID: \"96b454bc-04a9-4a82-ae03-94358fbe39c4\") "
Jul 24 01:00:51.403647 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:51.403621    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b454bc-04a9-4a82-ae03-94358fbe39c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96b454bc-04a9-4a82-ae03-94358fbe39c4" (UID: "96b454bc-04a9-4a82-ae03-94358fbe39c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:00:51.504090 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:51.504060    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b454bc-04a9-4a82-ae03-94358fbe39c4-maas-api-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:00:52.167640 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:52.167607    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"
Jul 24 01:00:52.167640 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:52.167630    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh" event={"ID":"96b454bc-04a9-4a82-ae03-94358fbe39c4","Type":"ContainerDied","Data":"4d20805e871c5dcbbf6a349c87d7d4199a782b0b74f9d1981f5d2c0570f8f370"}
Jul 24 01:00:52.167865 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:00:52.167682    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d20805e871c5dcbbf6a349c87d7d4199a782b0b74f9d1981f5d2c0570f8f370"
Jul 24 01:01:04.378513 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.378474    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"]
Jul 24 01:01:04.379020 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.378770    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv" podUID="69ff3965-b294-4e4a-8a6d-e6bcf4ac6886" containerName="maas-api" containerID="cri-o://b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68" gracePeriod=30
Jul 24 01:01:04.412947 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.412919    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms"]
Jul 24 01:01:04.416090 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.416066    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a3cf0e-29747580-mgrms"]
Jul 24 01:01:04.637481 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.637408    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 01:01:04.723919 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.723883    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-openshift-service-ca\") pod \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") "
Jul 24 01:01:04.724146 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.723944    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-maas-api-tls\") pod \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") "
Jul 24 01:01:04.724146 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.724021    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jrl9\" (UniqueName: \"kubernetes.io/projected/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-kube-api-access-4jrl9\") pod \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\" (UID: \"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886\") "
Jul 24 01:01:04.724346 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.724323    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69ff3965-b294-4e4a-8a6d-e6bcf4ac6886" (UID: "69ff3965-b294-4e4a-8a6d-e6bcf4ac6886"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:01:04.726167 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.726137    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-kube-api-access-4jrl9" (OuterVolumeSpecName: "kube-api-access-4jrl9") pod "69ff3965-b294-4e4a-8a6d-e6bcf4ac6886" (UID: "69ff3965-b294-4e4a-8a6d-e6bcf4ac6886"). InnerVolumeSpecName "kube-api-access-4jrl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:01:04.726271 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.726220    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69ff3965-b294-4e4a-8a6d-e6bcf4ac6886" (UID: "69ff3965-b294-4e4a-8a6d-e6bcf4ac6886"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:01:04.825606 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.825570    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrl9\" (UniqueName: \"kubernetes.io/projected/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-kube-api-access-4jrl9\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:01:04.825606 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.825602    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:01:04.825606 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:04.825613    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:01:05.233651 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:05.233612    2560 generic.go:356] "Generic (PLEG): container finished" podID="69ff3965-b294-4e4a-8a6d-e6bcf4ac6886" containerID="b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68" exitCode=0
Jul 24 01:01:05.233816 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:05.233661    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"
Jul 24 01:01:05.233816 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:05.233699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv" event={"ID":"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886","Type":"ContainerDied","Data":"b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68"}
Jul 24 01:01:05.233816 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:05.233755    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv" event={"ID":"69ff3965-b294-4e4a-8a6d-e6bcf4ac6886","Type":"ContainerDied","Data":"3f6390bf5943142b542fa9fd4ab48e7dac5b56a6933df2e0e74fed92983bd8c1"}
Jul 24 01:01:05.233816 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:05.233778    2560 scope.go:117] "RemoveContainer" containerID="b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68"
Jul 24 01:01:05.244033 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:05.244013    2560 scope.go:117] "RemoveContainer" containerID="b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68"
Jul 24 01:01:05.244310 ip-10-0-135-176 kubenswrapper[2560]: E0724 01:01:05.244293    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68\": container with ID starting with b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68 not found: ID does not exist" containerID="b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68"
Jul 24 01:01:05.244392 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:05.244317    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68"} err="failed to get container status \"b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68\": rpc error: code = NotFound desc = could not find container \"b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68\": container with ID starting with b2c79375d58405ac0044ab7b5d0ada1b3769fbdbafbebeb964a32dd0c9bfdf68 not found: ID does not exist"
Jul 24 01:01:05.254625 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:05.254595    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"]
Jul 24 01:01:05.259137 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:05.259115    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3cf0e-66b9596ffc-4n6hv"]
Jul 24 01:01:05.968028 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:05.967996    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69ff3965-b294-4e4a-8a6d-e6bcf4ac6886" path="/var/lib/kubelet/pods/69ff3965-b294-4e4a-8a6d-e6bcf4ac6886/volumes"
Jul 24 01:01:05.968380 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:05.968290    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca9f9a8e-ffa7-4aa8-adc5-0e1397421db3" path="/var/lib/kubelet/pods/ca9f9a8e-ffa7-4aa8-adc5-0e1397421db3/volumes"
Jul 24 01:01:08.951112 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:08.951072    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"]
Jul 24 01:01:08.956353 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:08.956326    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a3cf0e-qwqfh"]
Jul 24 01:01:09.968109 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:01:09.968065    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b454bc-04a9-4a82-ae03-94358fbe39c4" path="/var/lib/kubelet/pods/96b454bc-04a9-4a82-ae03-94358fbe39c4/volumes"
Jul 24 01:02:08.262462 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.262423    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"]
Jul 24 01:02:08.263018 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.262656    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs" podUID="9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b" containerName="maas-api" containerID="cri-o://d6c752f6d551388bc9674ab1c2e3fb424c19f83894470dc09b13fa4f1be103a2" gracePeriod=30
Jul 24 01:02:08.295808 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.295774    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557"]
Jul 24 01:02:08.298896 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.298870    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86b8da-29747580-zn557"]
Jul 24 01:02:08.331389 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.331359    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"]
Jul 24 01:02:08.331667 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.331614    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v" podUID="1ac9eb27-e91d-4b67-aa82-c77d3d175780" containerName="payload-pre-processing" containerID="cri-o://cabfcfa86f7eb4bb7013e1a1905b81f9703ceef378cfd55b858788674d997863" gracePeriod=30
Jul 24 01:02:08.553195 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.553164    2560 generic.go:356] "Generic (PLEG): container finished" podID="9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b" containerID="d6c752f6d551388bc9674ab1c2e3fb424c19f83894470dc09b13fa4f1be103a2" exitCode=0
Jul 24 01:02:08.553357 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.553240    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs" event={"ID":"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b","Type":"ContainerDied","Data":"d6c752f6d551388bc9674ab1c2e3fb424c19f83894470dc09b13fa4f1be103a2"}
Jul 24 01:02:08.555288 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.555259    2560 generic.go:356] "Generic (PLEG): container finished" podID="1ac9eb27-e91d-4b67-aa82-c77d3d175780" containerID="cabfcfa86f7eb4bb7013e1a1905b81f9703ceef378cfd55b858788674d997863" exitCode=0
Jul 24 01:02:08.555405 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.555329    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v" event={"ID":"1ac9eb27-e91d-4b67-aa82-c77d3d175780","Type":"ContainerDied","Data":"cabfcfa86f7eb4bb7013e1a1905b81f9703ceef378cfd55b858788674d997863"}
Jul 24 01:02:08.584282 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.584257    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 01:02:08.596789 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.596759    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-openshift-service-ca\") pod \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") "
Jul 24 01:02:08.596945 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.596873    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-maas-api-tls\") pod \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") "
Jul 24 01:02:08.596945 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.596915    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zld4g\" (UniqueName: \"kubernetes.io/projected/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-kube-api-access-zld4g\") pod \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\" (UID: \"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b\") "
Jul 24 01:02:08.597243 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.597191    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b" (UID: "9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:02:08.599159 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.599125    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-kube-api-access-zld4g" (OuterVolumeSpecName: "kube-api-access-zld4g") pod "9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b" (UID: "9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b"). InnerVolumeSpecName "kube-api-access-zld4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:02:08.599279 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.599160    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b" (UID: "9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:02:08.621493 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.621468    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 01:02:08.698240 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.698210    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lwnb\" (UniqueName: \"kubernetes.io/projected/1ac9eb27-e91d-4b67-aa82-c77d3d175780-kube-api-access-9lwnb\") pod \"1ac9eb27-e91d-4b67-aa82-c77d3d175780\" (UID: \"1ac9eb27-e91d-4b67-aa82-c77d3d175780\") "
Jul 24 01:02:08.698402 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.698298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac9eb27-e91d-4b67-aa82-c77d3d175780-plugins-config-volume\") pod \"1ac9eb27-e91d-4b67-aa82-c77d3d175780\" (UID: \"1ac9eb27-e91d-4b67-aa82-c77d3d175780\") "
Jul 24 01:02:08.698601 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.698577    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:08.698703 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.698603    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:08.698703 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.698629    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zld4g\" (UniqueName: \"kubernetes.io/projected/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b-kube-api-access-zld4g\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:08.698703 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.698651    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac9eb27-e91d-4b67-aa82-c77d3d175780-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ac9eb27-e91d-4b67-aa82-c77d3d175780" (UID: "1ac9eb27-e91d-4b67-aa82-c77d3d175780"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:02:08.700284 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.700267    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac9eb27-e91d-4b67-aa82-c77d3d175780-kube-api-access-9lwnb" (OuterVolumeSpecName: "kube-api-access-9lwnb") pod "1ac9eb27-e91d-4b67-aa82-c77d3d175780" (UID: "1ac9eb27-e91d-4b67-aa82-c77d3d175780"). InnerVolumeSpecName "kube-api-access-9lwnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:02:08.799802 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.799728    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac9eb27-e91d-4b67-aa82-c77d3d175780-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:08.799802 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:08.799757    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lwnb\" (UniqueName: \"kubernetes.io/projected/1ac9eb27-e91d-4b67-aa82-c77d3d175780-kube-api-access-9lwnb\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:09.562331 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.562246    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"
Jul 24 01:02:09.562331 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.562259    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs" event={"ID":"9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b","Type":"ContainerDied","Data":"80a70fd6af11b1a5c6bae29127a3a598ad009091baa8d57bdb61ca58d1caf40d"}
Jul 24 01:02:09.562856 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.562331    2560 scope.go:117] "RemoveContainer" containerID="d6c752f6d551388bc9674ab1c2e3fb424c19f83894470dc09b13fa4f1be103a2"
Jul 24 01:02:09.563895 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.563864    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v" event={"ID":"1ac9eb27-e91d-4b67-aa82-c77d3d175780","Type":"ContainerDied","Data":"479cd808d1c2dd8064fcf559b4043447b7d29f1a514a7807762689273c178a11"}
Jul 24 01:02:09.564039 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.563898    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"
Jul 24 01:02:09.573286 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.573257    2560 scope.go:117] "RemoveContainer" containerID="cabfcfa86f7eb4bb7013e1a1905b81f9703ceef378cfd55b858788674d997863"
Jul 24 01:02:09.588549 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.588522    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"]
Jul 24 01:02:09.591774 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.591752    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86b8da-dffd6f9d7-tjfhs"]
Jul 24 01:02:09.601066 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.601040    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"]
Jul 24 01:02:09.605080 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.605058    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86b8da-6f8cfb8d56-f742v"]
Jul 24 01:02:09.968347 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.968304    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ac9eb27-e91d-4b67-aa82-c77d3d175780" path="/var/lib/kubelet/pods/1ac9eb27-e91d-4b67-aa82-c77d3d175780/volumes"
Jul 24 01:02:09.968688 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.968667    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47db328d-d561-47d7-8c25-a3a7bc9c6c06" path="/var/lib/kubelet/pods/47db328d-d561-47d7-8c25-a3a7bc9c6c06/volumes"
Jul 24 01:02:09.969080 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:09.969061    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b" path="/var/lib/kubelet/pods/9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b/volumes"
Jul 24 01:02:30.308384 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.308344    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"]
Jul 24 01:02:30.308994 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.308587    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh" podUID="76848b08-90c1-4723-85c5-3a876bb02b0e" containerName="maas-api" containerID="cri-o://752a05723bf0e6e891c8bdfe89c34bdaf38c6c76def985025bec0db2a68d57f2" gracePeriod=30
Jul 24 01:02:30.351142 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.351107    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2"]
Jul 24 01:02:30.354699 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.354675    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-38da25-29747580-9r9b2"]
Jul 24 01:02:30.379364 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.379335    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"]
Jul 24 01:02:30.379568 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.379543    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l" podUID="7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236" containerName="payload-processing" containerID="cri-o://e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985" gracePeriod=30
Jul 24 01:02:30.389956 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.388085    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"]
Jul 24 01:02:30.389956 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.388324    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8" podUID="a1bb8779-2fcd-4929-b49a-d3105bdce1e2" containerName="payload-pre-processing" containerID="cri-o://ba664aec1a017a385429bafdf4118c26d071df40967e43b8669d8f6f7ecb4b8b" gracePeriod=30
Jul 24 01:02:30.674605 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.674570    2560 generic.go:356] "Generic (PLEG): container finished" podID="76848b08-90c1-4723-85c5-3a876bb02b0e" containerID="752a05723bf0e6e891c8bdfe89c34bdaf38c6c76def985025bec0db2a68d57f2" exitCode=0
Jul 24 01:02:30.674782 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.674670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh" event={"ID":"76848b08-90c1-4723-85c5-3a876bb02b0e","Type":"ContainerDied","Data":"752a05723bf0e6e891c8bdfe89c34bdaf38c6c76def985025bec0db2a68d57f2"}
Jul 24 01:02:30.676430 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.676408    2560 generic.go:356] "Generic (PLEG): container finished" podID="a1bb8779-2fcd-4929-b49a-d3105bdce1e2" containerID="ba664aec1a017a385429bafdf4118c26d071df40967e43b8669d8f6f7ecb4b8b" exitCode=0
Jul 24 01:02:30.676542 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.676445    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8" event={"ID":"a1bb8779-2fcd-4929-b49a-d3105bdce1e2","Type":"ContainerDied","Data":"ba664aec1a017a385429bafdf4118c26d071df40967e43b8669d8f6f7ecb4b8b"}
Jul 24 01:02:30.677091 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.677066    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 01:02:30.677784 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.677762    2560 generic.go:356] "Generic (PLEG): container finished" podID="7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236" containerID="e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985" exitCode=0
Jul 24 01:02:30.677897 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.677823    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l" event={"ID":"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236","Type":"ContainerDied","Data":"e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985"}
Jul 24 01:02:30.677973 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.677899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l" event={"ID":"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236","Type":"ContainerDied","Data":"99009c5ffe134514adb914ce63156551ae9eeedf228937b31fc8bbe393126ef5"}
Jul 24 01:02:30.677973 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.677932    2560 scope.go:117] "RemoveContainer" containerID="e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985"
Jul 24 01:02:30.680735 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.680716    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 01:02:30.687314 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.687291    2560 scope.go:117] "RemoveContainer" containerID="e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985"
Jul 24 01:02:30.695272 ip-10-0-135-176 kubenswrapper[2560]: E0724 01:02:30.695241    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985\": container with ID starting with e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985 not found: ID does not exist" containerID="e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985"
Jul 24 01:02:30.695395 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.695280    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985"} err="failed to get container status \"e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985\": rpc error: code = NotFound desc = could not find container \"e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985\": container with ID starting with e06142dd331e46c23eba8ed25e65939358b4671b45d956bd451d99b7970b7985 not found: ID does not exist"
Jul 24 01:02:30.699160 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.699134    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 01:02:30.790234 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.790204    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28thv\" (UniqueName: \"kubernetes.io/projected/76848b08-90c1-4723-85c5-3a876bb02b0e-kube-api-access-28thv\") pod \"76848b08-90c1-4723-85c5-3a876bb02b0e\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") "
Jul 24 01:02:30.790441 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.790241    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76848b08-90c1-4723-85c5-3a876bb02b0e-openshift-service-ca\") pod \"76848b08-90c1-4723-85c5-3a876bb02b0e\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") "
Jul 24 01:02:30.790441 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.790260    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh29z\" (UniqueName: \"kubernetes.io/projected/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-kube-api-access-jh29z\") pod \"a1bb8779-2fcd-4929-b49a-d3105bdce1e2\" (UID: \"a1bb8779-2fcd-4929-b49a-d3105bdce1e2\") "
Jul 24 01:02:30.790441 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.790294    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76848b08-90c1-4723-85c5-3a876bb02b0e-maas-api-tls\") pod \"76848b08-90c1-4723-85c5-3a876bb02b0e\" (UID: \"76848b08-90c1-4723-85c5-3a876bb02b0e\") "
Jul 24 01:02:30.790441 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.790406    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvwx2\" (UniqueName: \"kubernetes.io/projected/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-kube-api-access-jvwx2\") pod \"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236\" (UID: \"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236\") "
Jul 24 01:02:30.790677 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.790466    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-plugins-config-volume\") pod \"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236\" (UID: \"7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236\") "
Jul 24 01:02:30.790790 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.790763    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76848b08-90c1-4723-85c5-3a876bb02b0e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76848b08-90c1-4723-85c5-3a876bb02b0e" (UID: "76848b08-90c1-4723-85c5-3a876bb02b0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:02:30.790897 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.790879    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-plugins-config-volume\") pod \"a1bb8779-2fcd-4929-b49a-d3105bdce1e2\" (UID: \"a1bb8779-2fcd-4929-b49a-d3105bdce1e2\") "
Jul 24 01:02:30.790973 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.790895    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236" (UID: "7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:02:30.791310 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.791285    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:30.791423 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.791312    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76848b08-90c1-4723-85c5-3a876bb02b0e-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:30.791423 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.791387    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1bb8779-2fcd-4929-b49a-d3105bdce1e2" (UID: "a1bb8779-2fcd-4929-b49a-d3105bdce1e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:02:30.792769 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.792733    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76848b08-90c1-4723-85c5-3a876bb02b0e-kube-api-access-28thv" (OuterVolumeSpecName: "kube-api-access-28thv") pod "76848b08-90c1-4723-85c5-3a876bb02b0e" (UID: "76848b08-90c1-4723-85c5-3a876bb02b0e"). InnerVolumeSpecName "kube-api-access-28thv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:02:30.792991 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.792974    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76848b08-90c1-4723-85c5-3a876bb02b0e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76848b08-90c1-4723-85c5-3a876bb02b0e" (UID: "76848b08-90c1-4723-85c5-3a876bb02b0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:02:30.793043 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.792998    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-kube-api-access-jvwx2" (OuterVolumeSpecName: "kube-api-access-jvwx2") pod "7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236" (UID: "7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236"). InnerVolumeSpecName "kube-api-access-jvwx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:02:30.793080 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.793060    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-kube-api-access-jh29z" (OuterVolumeSpecName: "kube-api-access-jh29z") pod "a1bb8779-2fcd-4929-b49a-d3105bdce1e2" (UID: "a1bb8779-2fcd-4929-b49a-d3105bdce1e2"). InnerVolumeSpecName "kube-api-access-jh29z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:02:30.891988 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.891886    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28thv\" (UniqueName: \"kubernetes.io/projected/76848b08-90c1-4723-85c5-3a876bb02b0e-kube-api-access-28thv\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:30.891988 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.891922    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh29z\" (UniqueName: \"kubernetes.io/projected/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-kube-api-access-jh29z\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:30.891988 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.891932    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76848b08-90c1-4723-85c5-3a876bb02b0e-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:30.891988 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.891941    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvwx2\" (UniqueName: \"kubernetes.io/projected/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236-kube-api-access-jvwx2\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:30.891988 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:30.891951    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bb8779-2fcd-4929-b49a-d3105bdce1e2-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:31.684267 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.684172    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"
Jul 24 01:02:31.684267 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.684184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh" event={"ID":"76848b08-90c1-4723-85c5-3a876bb02b0e","Type":"ContainerDied","Data":"3c30fb315b9938cab6a45459da5d3f3c971854f3e7754646ce5f4f0b4833300c"}
Jul 24 01:02:31.684267 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.684260    2560 scope.go:117] "RemoveContainer" containerID="752a05723bf0e6e891c8bdfe89c34bdaf38c6c76def985025bec0db2a68d57f2"
Jul 24 01:02:31.685555 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.685541    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"
Jul 24 01:02:31.685649 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.685579    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8" event={"ID":"a1bb8779-2fcd-4929-b49a-d3105bdce1e2","Type":"ContainerDied","Data":"1f2274143d79c9a798769f01ec79bb4c83c056fdcff7136fad72811e10c23bdb"}
Jul 24 01:02:31.686660 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.686629    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"
Jul 24 01:02:31.694956 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.694934    2560 scope.go:117] "RemoveContainer" containerID="ba664aec1a017a385429bafdf4118c26d071df40967e43b8669d8f6f7ecb4b8b"
Jul 24 01:02:31.711815 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.711788    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"]
Jul 24 01:02:31.715879 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.715853    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-38da25-64565dd698-k5fdh"]
Jul 24 01:02:31.726532 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.726509    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"]
Jul 24 01:02:31.730236 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.730205    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-38da25-55f947f696-k6mp8"]
Jul 24 01:02:31.740714 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.740690    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"]
Jul 24 01:02:31.745328 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.745291    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-38da25-fd86b95bb-zv77l"]
Jul 24 01:02:31.967956 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.967921    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76848b08-90c1-4723-85c5-3a876bb02b0e" path="/var/lib/kubelet/pods/76848b08-90c1-4723-85c5-3a876bb02b0e/volumes"
Jul 24 01:02:31.968281 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.968267    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236" path="/var/lib/kubelet/pods/7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236/volumes"
Jul 24 01:02:31.968575 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.968563    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1bb8779-2fcd-4929-b49a-d3105bdce1e2" path="/var/lib/kubelet/pods/a1bb8779-2fcd-4929-b49a-d3105bdce1e2/volumes"
Jul 24 01:02:31.968793 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:31.968781    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b895a037-979a-4336-9891-99b3ef6de8fb" path="/var/lib/kubelet/pods/b895a037-979a-4336-9891-99b3ef6de8fb/volumes"
Jul 24 01:02:42.134620 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.134583    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"]
Jul 24 01:02:42.135201 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135179    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b454bc-04a9-4a82-ae03-94358fbe39c4" containerName="revoke-keys"
Jul 24 01:02:42.135292 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135204    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b454bc-04a9-4a82-ae03-94358fbe39c4" containerName="revoke-keys"
Jul 24 01:02:42.135292 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135224    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1bb8779-2fcd-4929-b49a-d3105bdce1e2" containerName="payload-pre-processing"
Jul 24 01:02:42.135292 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135232    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1bb8779-2fcd-4929-b49a-d3105bdce1e2" containerName="payload-pre-processing"
Jul 24 01:02:42.135292 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135247    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ff3965-b294-4e4a-8a6d-e6bcf4ac6886" containerName="maas-api"
Jul 24 01:02:42.135292 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135256    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ff3965-b294-4e4a-8a6d-e6bcf4ac6886" containerName="maas-api"
Jul 24 01:02:42.135292 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135270    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236" containerName="payload-processing"
Jul 24 01:02:42.135292 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135279    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236" containerName="payload-processing"
Jul 24 01:02:42.135292 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135288    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b" containerName="maas-api"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135297    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b" containerName="maas-api"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135315    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76848b08-90c1-4723-85c5-3a876bb02b0e" containerName="maas-api"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135324    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="76848b08-90c1-4723-85c5-3a876bb02b0e" containerName="maas-api"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135340    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac9eb27-e91d-4b67-aa82-c77d3d175780" containerName="payload-pre-processing"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135349    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac9eb27-e91d-4b67-aa82-c77d3d175780" containerName="payload-pre-processing"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135454    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ff3965-b294-4e4a-8a6d-e6bcf4ac6886" containerName="maas-api"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135469    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1bb8779-2fcd-4929-b49a-d3105bdce1e2" containerName="payload-pre-processing"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135494    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac9eb27-e91d-4b67-aa82-c77d3d175780" containerName="payload-pre-processing"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135504    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b454bc-04a9-4a82-ae03-94358fbe39c4" containerName="revoke-keys"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135515    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc9d9b8-5fce-46c6-8e2f-65ac22bb3236" containerName="payload-processing"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135524    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b24ae48-b86a-4b5c-80cb-9c8c6220cf5b" containerName="maas-api"
Jul 24 01:02:42.135687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.135535    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="76848b08-90c1-4723-85c5-3a876bb02b0e" containerName="maas-api"
Jul 24 01:02:42.142622 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.142577    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"
Jul 24 01:02:42.146076 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.146049    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rz5kk\""
Jul 24 01:02:42.146822 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.146742    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"]
Jul 24 01:02:42.283235 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.283194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d29cda8-b049-4f05-8e28-e047e2cd548f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg\" (UID: \"2d29cda8-b049-4f05-8e28-e047e2cd548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"
Jul 24 01:02:42.384719 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.384630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d29cda8-b049-4f05-8e28-e047e2cd548f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg\" (UID: \"2d29cda8-b049-4f05-8e28-e047e2cd548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"
Jul 24 01:02:42.385231 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.385208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d29cda8-b049-4f05-8e28-e047e2cd548f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg\" (UID: \"2d29cda8-b049-4f05-8e28-e047e2cd548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"
Jul 24 01:02:42.456685 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.456650    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"
Jul 24 01:02:42.593653 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.593623    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"]
Jul 24 01:02:42.596075 ip-10-0-135-176 kubenswrapper[2560]: W0724 01:02:42.596043    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d29cda8_b049_4f05_8e28_e047e2cd548f.slice/crio-c054d46a2e371a872c879135142b8b8c64ecd36c3c89ca2cb3cfff0d79634906 WatchSource:0}: Error finding container c054d46a2e371a872c879135142b8b8c64ecd36c3c89ca2cb3cfff0d79634906: Status 404 returned error can't find the container with id c054d46a2e371a872c879135142b8b8c64ecd36c3c89ca2cb3cfff0d79634906
Jul 24 01:02:42.744189 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:42.744153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg" event={"ID":"2d29cda8-b049-4f05-8e28-e047e2cd548f","Type":"ContainerStarted","Data":"c054d46a2e371a872c879135142b8b8c64ecd36c3c89ca2cb3cfff0d79634906"}
Jul 24 01:02:43.751025 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:43.750982    2560 generic.go:356] "Generic (PLEG): container finished" podID="2d29cda8-b049-4f05-8e28-e047e2cd548f" containerID="5a900feb0b9dc729a79e837d80cbab5970bece349dcf08de46b2d0eb3311db29" exitCode=0
Jul 24 01:02:43.751444 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:43.751073    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg" event={"ID":"2d29cda8-b049-4f05-8e28-e047e2cd548f","Type":"ContainerDied","Data":"5a900feb0b9dc729a79e837d80cbab5970bece349dcf08de46b2d0eb3311db29"}
Jul 24 01:02:44.889262 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:44.889230    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"
Jul 24 01:02:45.008348 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:45.008260    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d29cda8-b049-4f05-8e28-e047e2cd548f-maas-api-service-ca\") pod \"2d29cda8-b049-4f05-8e28-e047e2cd548f\" (UID: \"2d29cda8-b049-4f05-8e28-e047e2cd548f\") "
Jul 24 01:02:45.008720 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:45.008696    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d29cda8-b049-4f05-8e28-e047e2cd548f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d29cda8-b049-4f05-8e28-e047e2cd548f" (UID: "2d29cda8-b049-4f05-8e28-e047e2cd548f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:02:45.109628 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:45.109589    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d29cda8-b049-4f05-8e28-e047e2cd548f-maas-api-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:45.762755 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:45.762717    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg" event={"ID":"2d29cda8-b049-4f05-8e28-e047e2cd548f","Type":"ContainerDied","Data":"c054d46a2e371a872c879135142b8b8c64ecd36c3c89ca2cb3cfff0d79634906"}
Jul 24 01:02:45.763000 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:45.762763    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"
Jul 24 01:02:45.763000 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:45.762768    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c054d46a2e371a872c879135142b8b8c64ecd36c3c89ca2cb3cfff0d79634906"
Jul 24 01:02:52.713080 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:52.713039    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"]
Jul 24 01:02:52.713548 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:52.713288    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6" podUID="1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1" containerName="maas-api" containerID="cri-o://4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa" gracePeriod=30
Jul 24 01:02:52.776746 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:52.776709    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"]
Jul 24 01:02:52.776960 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:52.776933    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l" podUID="f6f5cef3-5b81-49bc-9c95-ac4771978b7f" containerName="payload-processing" containerID="cri-o://73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88" gracePeriod=30
Jul 24 01:02:52.784765 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:52.784736    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"]
Jul 24 01:02:52.785032 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:52.784983    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t" podUID="6dc58066-e31c-452b-acef-b63dda1465f1" containerName="payload-pre-processing" containerID="cri-o://a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f" gracePeriod=30
Jul 24 01:02:53.101672 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.101648    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 01:02:53.104648 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.104626    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 01:02:53.108120 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.108098    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 01:02:53.183132 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.183100    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-plugins-config-volume\") pod \"f6f5cef3-5b81-49bc-9c95-ac4771978b7f\" (UID: \"f6f5cef3-5b81-49bc-9c95-ac4771978b7f\") "
Jul 24 01:02:53.183279 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.183146    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p7t5\" (UniqueName: \"kubernetes.io/projected/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-kube-api-access-8p7t5\") pod \"f6f5cef3-5b81-49bc-9c95-ac4771978b7f\" (UID: \"f6f5cef3-5b81-49bc-9c95-ac4771978b7f\") "
Jul 24 01:02:53.183495 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.183468    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6f5cef3-5b81-49bc-9c95-ac4771978b7f" (UID: "f6f5cef3-5b81-49bc-9c95-ac4771978b7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:02:53.185186 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.185167    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-kube-api-access-8p7t5" (OuterVolumeSpecName: "kube-api-access-8p7t5") pod "f6f5cef3-5b81-49bc-9c95-ac4771978b7f" (UID: "f6f5cef3-5b81-49bc-9c95-ac4771978b7f"). InnerVolumeSpecName "kube-api-access-8p7t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:02:53.284668 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.284587    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsn24\" (UniqueName: \"kubernetes.io/projected/6dc58066-e31c-452b-acef-b63dda1465f1-kube-api-access-zsn24\") pod \"6dc58066-e31c-452b-acef-b63dda1465f1\" (UID: \"6dc58066-e31c-452b-acef-b63dda1465f1\") "
Jul 24 01:02:53.284668 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.284637    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-maas-api-tls\") pod \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") "
Jul 24 01:02:53.284899 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.284786    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc58066-e31c-452b-acef-b63dda1465f1-plugins-config-volume\") pod \"6dc58066-e31c-452b-acef-b63dda1465f1\" (UID: \"6dc58066-e31c-452b-acef-b63dda1465f1\") "
Jul 24 01:02:53.284899 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.284821    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-openshift-service-ca\") pod \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") "
Jul 24 01:02:53.284899 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.284888    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts9g5\" (UniqueName: \"kubernetes.io/projected/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-kube-api-access-ts9g5\") pod \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\" (UID: \"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1\") "
Jul 24 01:02:53.285154 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.285134    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:53.285251 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.285158    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p7t5\" (UniqueName: \"kubernetes.io/projected/f6f5cef3-5b81-49bc-9c95-ac4771978b7f-kube-api-access-8p7t5\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:53.285251 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.285179    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1" (UID: "1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:02:53.285251 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.285191    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc58066-e31c-452b-acef-b63dda1465f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dc58066-e31c-452b-acef-b63dda1465f1" (UID: "6dc58066-e31c-452b-acef-b63dda1465f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:02:53.286847 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.286807    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1" (UID: "1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:02:53.286939 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.286806    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-kube-api-access-ts9g5" (OuterVolumeSpecName: "kube-api-access-ts9g5") pod "1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1" (UID: "1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1"). InnerVolumeSpecName "kube-api-access-ts9g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:02:53.286939 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.286903    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc58066-e31c-452b-acef-b63dda1465f1-kube-api-access-zsn24" (OuterVolumeSpecName: "kube-api-access-zsn24") pod "6dc58066-e31c-452b-acef-b63dda1465f1" (UID: "6dc58066-e31c-452b-acef-b63dda1465f1"). InnerVolumeSpecName "kube-api-access-zsn24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:02:53.385918 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.385883    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc58066-e31c-452b-acef-b63dda1465f1-plugins-config-volume\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:53.385918 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.385915    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-openshift-service-ca\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:53.385918 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.385924    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts9g5\" (UniqueName: \"kubernetes.io/projected/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-kube-api-access-ts9g5\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:53.386148 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.385933    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsn24\" (UniqueName: \"kubernetes.io/projected/6dc58066-e31c-452b-acef-b63dda1465f1-kube-api-access-zsn24\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:53.386148 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.385944    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1-maas-api-tls\") on node \"ip-10-0-135-176.ec2.internal\" DevicePath \"\""
Jul 24 01:02:53.807659 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.807625    2560 generic.go:356] "Generic (PLEG): container finished" podID="6dc58066-e31c-452b-acef-b63dda1465f1" containerID="a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f" exitCode=0
Jul 24 01:02:53.808107 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.807668    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"
Jul 24 01:02:53.808107 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.807709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t" event={"ID":"6dc58066-e31c-452b-acef-b63dda1465f1","Type":"ContainerDied","Data":"a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f"}
Jul 24 01:02:53.808107 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.807772    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t" event={"ID":"6dc58066-e31c-452b-acef-b63dda1465f1","Type":"ContainerDied","Data":"1c45814efcaa02307a5809ac988f04ba1f9764d3d35f9e513bbd182919ae05a7"}
Jul 24 01:02:53.808107 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.807805    2560 scope.go:117] "RemoveContainer" containerID="a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f"
Jul 24 01:02:53.808988 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.808951    2560 generic.go:356] "Generic (PLEG): container finished" podID="f6f5cef3-5b81-49bc-9c95-ac4771978b7f" containerID="73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88" exitCode=0
Jul 24 01:02:53.808988 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.808981    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l" event={"ID":"f6f5cef3-5b81-49bc-9c95-ac4771978b7f","Type":"ContainerDied","Data":"73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88"}
Jul 24 01:02:53.809211 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.809001    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"
Jul 24 01:02:53.809211 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.809025    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l" event={"ID":"f6f5cef3-5b81-49bc-9c95-ac4771978b7f","Type":"ContainerDied","Data":"cbae8e2e0e0b8cf1ddf201bf6bad3c8cb9b58230e6a013a86113be5be7d3b43c"}
Jul 24 01:02:53.820666 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.810878    2560 generic.go:356] "Generic (PLEG): container finished" podID="1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1" containerID="4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa" exitCode=0
Jul 24 01:02:53.820666 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.810922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6" event={"ID":"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1","Type":"ContainerDied","Data":"4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa"}
Jul 24 01:02:53.820666 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.810952    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"
Jul 24 01:02:53.820666 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.810984    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6" event={"ID":"1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1","Type":"ContainerDied","Data":"92782ef7ad5e2d07960cef1abe4fc6d9f461ce41fcfe103a84bb56fe9a8486d1"}
Jul 24 01:02:53.821439 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.821217    2560 scope.go:117] "RemoveContainer" containerID="a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f"
Jul 24 01:02:53.821641 ip-10-0-135-176 kubenswrapper[2560]: E0724 01:02:53.821610    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f\": container with ID starting with a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f not found: ID does not exist" containerID="a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f"
Jul 24 01:02:53.821730 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.821652    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f"} err="failed to get container status \"a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f\": rpc error: code = NotFound desc = could not find container \"a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f\": container with ID starting with a7c12981e80833c495882fc427c7bc807543bb0f514d3a5e9b3785ca987a890f not found: ID does not exist"
Jul 24 01:02:53.821730 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.821675    2560 scope.go:117] "RemoveContainer" containerID="73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88"
Jul 24 01:02:53.830892 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.830874    2560 scope.go:117] "RemoveContainer" containerID="73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88"
Jul 24 01:02:53.831161 ip-10-0-135-176 kubenswrapper[2560]: E0724 01:02:53.831142    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88\": container with ID starting with 73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88 not found: ID does not exist" containerID="73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88"
Jul 24 01:02:53.831214 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.831168    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88"} err="failed to get container status \"73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88\": rpc error: code = NotFound desc = could not find container \"73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88\": container with ID starting with 73f7276d8b4adf980de63c3a07b62db4b1854ea8c930a3558247a06c63d26c88 not found: ID does not exist"
Jul 24 01:02:53.831214 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.831186    2560 scope.go:117] "RemoveContainer" containerID="4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa"
Jul 24 01:02:53.835136 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.835116    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"]
Jul 24 01:02:53.838824 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.838797    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-44edfc-b4666cd66-kh59l"]
Jul 24 01:02:53.839415 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.839398    2560 scope.go:117] "RemoveContainer" containerID="4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa"
Jul 24 01:02:53.839732 ip-10-0-135-176 kubenswrapper[2560]: E0724 01:02:53.839707    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa\": container with ID starting with 4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa not found: ID does not exist" containerID="4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa"
Jul 24 01:02:53.839823 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.839734    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa"} err="failed to get container status \"4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa\": rpc error: code = NotFound desc = could not find container \"4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa\": container with ID starting with 4f8d9473c23dbb6ceda79576722e6cffc6f2eab1e61892e9f75830bcfd759ffa not found: ID does not exist"
Jul 24 01:02:53.848511 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.848487    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"]
Jul 24 01:02:53.852731 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.852700    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-44edfc-854f646f9b-qb22t"]
Jul 24 01:02:53.861520 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.861490    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"]
Jul 24 01:02:53.865443 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.865421    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-44edfc-9484d88c4-pz6t6"]
Jul 24 01:02:53.967665 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.967632    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1" path="/var/lib/kubelet/pods/1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1/volumes"
Jul 24 01:02:53.968002 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.967988    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dc58066-e31c-452b-acef-b63dda1465f1" path="/var/lib/kubelet/pods/6dc58066-e31c-452b-acef-b63dda1465f1/volumes"
Jul 24 01:02:53.968300 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:53.968289    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6f5cef3-5b81-49bc-9c95-ac4771978b7f" path="/var/lib/kubelet/pods/f6f5cef3-5b81-49bc-9c95-ac4771978b7f/volumes"
Jul 24 01:02:57.211479 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:57.211429    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"]
Jul 24 01:02:57.213428 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:57.213400    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-44edfc-pbcfg"]
Jul 24 01:02:57.967662 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:02:57.967624    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d29cda8-b049-4f05-8e28-e047e2cd548f" path="/var/lib/kubelet/pods/2d29cda8-b049-4f05-8e28-e047e2cd548f/volumes"
Jul 24 01:03:04.450974 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:04.450890    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7df697bf54-5rxg7_bb06662a-e767-4fc3-a09b-b4eb104b2fbb/authorino/0.log"
Jul 24 01:03:10.004580 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:10.004489    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8pjnq_e9fdf508-6600-4330-bd5d-4bda72e7d1e5/manager/0.log"
Jul 24 01:03:10.128235 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:10.128199    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-x6c65_9c11c1dc-96a2-4767-ae53-2c7c468bc7d3/manager/0.log"
Jul 24 01:03:10.258540 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:10.258443    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d76449b94-7w7cr_9f943000-89ca-4640-8c6c-852915e4eb2d/manager/0.log"
Jul 24 01:03:10.378855 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:10.378808    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bhwn4_e25170aa-f54f-4cb0-9334-585e5e89d007/server/0.log"
Jul 24 01:03:10.503687 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:10.503649    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wg7tt_790e6f76-09fa-4d0f-ab8f-6abcd6f748cc/manager/0.log"
Jul 24 01:03:10.910950 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:10.910922    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cff767bd6-q7l64_5a3ded49-8443-40d7-83c2-f14adf4a2cc0/manager/0.log"
Jul 24 01:03:10.917315 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:10.917294    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cff767bd6-q7l64_5a3ded49-8443-40d7-83c2-f14adf4a2cc0/copy-manifests/0.log"
Jul 24 01:03:11.280260 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:11.280225    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5d6c55ddd9-vwtrh_ce3e0420-3fc5-4faa-be8a-5a59be301692/maas-api/0.log"
Jul 24 01:03:11.745187 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:11.745158    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-km5mh_ef89d7fd-4e6b-4660-b06d-edbd560b2ed4/postgres/0.log"
Jul 24 01:03:13.079558 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:13.079519    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7df697bf54-5rxg7_bb06662a-e767-4fc3-a09b-b4eb104b2fbb/authorino/0.log"
Jul 24 01:03:13.198463 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:13.198429    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gx4jf_ac0ceea5-a2aa-44c4-95d3-bc4862cad4f3/manager/0.log"
Jul 24 01:03:15.005342 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:15.005308    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-nw6g8_695ae2f8-1dcd-4555-b5b1-f683cacb89e4/payload-processing/0.log"
Jul 24 01:03:15.604642 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:15.604609    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-f47jj_fdae42e0-3b15-4eeb-a909-7ae17865ee24/main/0.log"
Jul 24 01:03:27.714461 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:27.714429    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kgsn8_71f46a43-3c16-45f8-90c4-0f815e6850d0/global-pull-secret-syncer/0.log"
Jul 24 01:03:27.883966 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:27.883929    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xctk4_b87ed47a-6f1c-4f68-ae51-11144c46d320/konnectivity-agent/0.log"
Jul 24 01:03:27.952038 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:27.952010    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-176.ec2.internal_18fde8ac8173b64a16e5b7537a21972f/haproxy/0.log"
Jul 24 01:03:33.446065 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:33.446026    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7df697bf54-5rxg7_bb06662a-e767-4fc3-a09b-b4eb104b2fbb/authorino/0.log"
Jul 24 01:03:33.475421 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:33.475384    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gx4jf_ac0ceea5-a2aa-44c4-95d3-bc4862cad4f3/manager/0.log"
Jul 24 01:03:35.821419 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:35.821384    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-8x9kb_cc5d1a31-2a44-4139-b308-bcf393724c12/cluster-monitoring-operator/0.log"
Jul 24 01:03:35.965699 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:35.965663    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bvnkw_015bd884-f690-4837-a093-b29203d0fd09/node-exporter/0.log"
Jul 24 01:03:35.994525 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:35.994498    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bvnkw_015bd884-f690-4837-a093-b29203d0fd09/kube-rbac-proxy/0.log"
Jul 24 01:03:36.017106 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:36.017079    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bvnkw_015bd884-f690-4837-a093-b29203d0fd09/init-textfile/0.log"
Jul 24 01:03:37.916044 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916014    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"]
Jul 24 01:03:37.916420 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916357    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1" containerName="maas-api"
Jul 24 01:03:37.916420 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916369    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1" containerName="maas-api"
Jul 24 01:03:37.916420 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916380    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f5cef3-5b81-49bc-9c95-ac4771978b7f" containerName="payload-processing"
Jul 24 01:03:37.916420 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916387    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f5cef3-5b81-49bc-9c95-ac4771978b7f" containerName="payload-processing"
Jul 24 01:03:37.916420 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916395    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc58066-e31c-452b-acef-b63dda1465f1" containerName="payload-pre-processing"
Jul 24 01:03:37.916420 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916400    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc58066-e31c-452b-acef-b63dda1465f1" containerName="payload-pre-processing"
Jul 24 01:03:37.916420 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916407    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d29cda8-b049-4f05-8e28-e047e2cd548f" containerName="revoke-keys"
Jul 24 01:03:37.916420 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916413    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d29cda8-b049-4f05-8e28-e047e2cd548f" containerName="revoke-keys"
Jul 24 01:03:37.916673 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916466    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac0cdc3-eb7b-4582-9bb7-d60998c1d5a1" containerName="maas-api"
Jul 24 01:03:37.916673 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916476    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d29cda8-b049-4f05-8e28-e047e2cd548f" containerName="revoke-keys"
Jul 24 01:03:37.916673 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916483    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f5cef3-5b81-49bc-9c95-ac4771978b7f" containerName="payload-processing"
Jul 24 01:03:37.916673 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.916488    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc58066-e31c-452b-acef-b63dda1465f1" containerName="payload-pre-processing"
Jul 24 01:03:37.919717 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.919696    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:37.921956 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.921935    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sslln\"/\"kube-root-ca.crt\""
Jul 24 01:03:37.922045 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.922031    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sslln\"/\"openshift-service-ca.crt\""
Jul 24 01:03:37.922693 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.922676    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sslln\"/\"default-dockercfg-qwwxr\""
Jul 24 01:03:37.929624 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:37.929598    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"]
Jul 24 01:03:38.060025 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.059989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-podres\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.060200 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.060032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-lib-modules\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.060200 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.060055    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rqg\" (UniqueName: \"kubernetes.io/projected/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-kube-api-access-68rqg\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.060200 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.060136    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-sys\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.060200 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.060181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-proc\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.161317 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.161282    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-podres\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.161506 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.161325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-lib-modules\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.161506 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.161347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rqg\" (UniqueName: \"kubernetes.io/projected/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-kube-api-access-68rqg\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.161506 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.161385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-sys\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.161506 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.161409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-proc\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.161506 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.161460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-podres\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.161506 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.161470    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-lib-modules\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.161506 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.161488    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-proc\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.161815 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.161510    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-sys\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.169792 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.169736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rqg\" (UniqueName: \"kubernetes.io/projected/b2ff9c7f-8c67-4c12-adc7-80e9f15c2621-kube-api-access-68rqg\") pod \"perf-node-gather-daemonset-2n2fg\" (UID: \"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621\") " pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.230911 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.230873    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:38.361414 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:38.361389    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"]
Jul 24 01:03:38.364056 ip-10-0-135-176 kubenswrapper[2560]: W0724 01:03:38.364025    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb2ff9c7f_8c67_4c12_adc7_80e9f15c2621.slice/crio-407c47480421d143475777604c7efe4b7a5d5fc5867663dd243178546c8428de WatchSource:0}: Error finding container 407c47480421d143475777604c7efe4b7a5d5fc5867663dd243178546c8428de: Status 404 returned error can't find the container with id 407c47480421d143475777604c7efe4b7a5d5fc5867663dd243178546c8428de
Jul 24 01:03:39.029103 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:39.029069    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg" event={"ID":"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621","Type":"ContainerStarted","Data":"fb4e29bfbf9884ffd327b5feef6d15c96fcbc52295f8b01ecd21ececfca58418"}
Jul 24 01:03:39.029476 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:39.029119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg" event={"ID":"b2ff9c7f-8c67-4c12-adc7-80e9f15c2621","Type":"ContainerStarted","Data":"407c47480421d143475777604c7efe4b7a5d5fc5867663dd243178546c8428de"}
Jul 24 01:03:39.029476 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:39.029145    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:39.044403 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:39.044336    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg" podStartSLOduration=2.04432192 podStartE2EDuration="2.04432192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:03:39.043309511 +0000 UTC m=+2999.611125662" watchObservedRunningTime="2026-07-24 01:03:39.04432192 +0000 UTC m=+2999.612138070"
Jul 24 01:03:40.729130 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:40.729094    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kfm7w_483be78f-e144-4a62-8932-88df4550a1af/dns/0.log"
Jul 24 01:03:40.751543 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:40.751515    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kfm7w_483be78f-e144-4a62-8932-88df4550a1af/kube-rbac-proxy/0.log"
Jul 24 01:03:40.918601 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:40.918566    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mttpm_a00cf2c5-5c20-410f-bae2-80170581b317/dns-node-resolver/0.log"
Jul 24 01:03:41.535115 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:41.535084    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kd9fr_6d50d211-9d46-4e14-ace4-554592a1ddc0/node-ca/0.log"
Jul 24 01:03:42.868198 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:42.868153    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-nw6g8_695ae2f8-1dcd-4555-b5b1-f683cacb89e4/payload-processing/0.log"
Jul 24 01:03:43.686119 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:43.686080    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7rbjj_ac29741e-a8ad-4a01-8342-e631c2eca6ce/serve-healthcheck-canary/0.log"
Jul 24 01:03:44.362929 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:44.362889    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w86dg_9da248de-af65-442a-9766-eae338141049/kube-rbac-proxy/0.log"
Jul 24 01:03:44.382038 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:44.382006    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w86dg_9da248de-af65-442a-9766-eae338141049/exporter/0.log"
Jul 24 01:03:44.403027 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:44.403003    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w86dg_9da248de-af65-442a-9766-eae338141049/extractor/0.log"
Jul 24 01:03:45.043968 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:45.043940    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sslln/perf-node-gather-daemonset-2n2fg"
Jul 24 01:03:46.586329 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:46.586293    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8pjnq_e9fdf508-6600-4330-bd5d-4bda72e7d1e5/manager/0.log"
Jul 24 01:03:46.932269 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:46.932243    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-x6c65_9c11c1dc-96a2-4767-ae53-2c7c468bc7d3/manager/0.log"
Jul 24 01:03:47.164763 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:47.164734    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d76449b94-7w7cr_9f943000-89ca-4640-8c6c-852915e4eb2d/manager/0.log"
Jul 24 01:03:47.186246 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:47.186174    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bhwn4_e25170aa-f54f-4cb0-9334-585e5e89d007/server/0.log"
Jul 24 01:03:47.307365 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:47.307330    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wg7tt_790e6f76-09fa-4d0f-ab8f-6abcd6f748cc/manager/0.log"
Jul 24 01:03:47.531106 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:47.531077    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cff767bd6-q7l64_5a3ded49-8443-40d7-83c2-f14adf4a2cc0/manager/0.log"
Jul 24 01:03:47.550690 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:47.550651    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cff767bd6-q7l64_5a3ded49-8443-40d7-83c2-f14adf4a2cc0/copy-manifests/0.log"
Jul 24 01:03:48.771781 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:48.771743    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-nw6g8_695ae2f8-1dcd-4555-b5b1-f683cacb89e4/payload-processing/0.log"
Jul 24 01:03:48.823383 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:48.823352    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b84c6bc85-px7mf_fb31fa45-e4d2-47a6-b259-9777f096d4fd/manager/0.log"
Jul 24 01:03:50.552301 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:50.552268    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-nw6g8_695ae2f8-1dcd-4555-b5b1-f683cacb89e4/payload-processing/0.log"
Jul 24 01:03:53.832677 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:53.832640    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-xhght_4e393c13-3ca6-4f6c-9687-8447041dd5f5/migrator/0.log"
Jul 24 01:03:53.853903 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:53.853874    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-xhght_4e393c13-3ca6-4f6c-9687-8447041dd5f5/graceful-termination/0.log"
Jul 24 01:03:54.228991 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:54.228956    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-jt449_5b5fb683-60bd-483c-a20b-51cd9a0285df/kube-storage-version-migrator-operator/1.log"
Jul 24 01:03:54.230653 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:54.230630    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-jt449_5b5fb683-60bd-483c-a20b-51cd9a0285df/kube-storage-version-migrator-operator/0.log"
Jul 24 01:03:55.406469 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:55.406432    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kw8qh_4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8/kube-multus-additional-cni-plugins/0.log"
Jul 24 01:03:55.428254 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:55.428225    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kw8qh_4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8/egress-router-binary-copy/0.log"
Jul 24 01:03:55.448725 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:55.448696    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kw8qh_4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8/cni-plugins/0.log"
Jul 24 01:03:55.469688 ip-10-0-135-176 kubenswrapper[2560]: I0724 01:03:55.469660    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kw8qh_4bf6f39f-3f1e-40d4-b5cc-2937ed06c2e8/bond-cni-plugin/0.log"