Jul 23 22:22:25.555284 ip-10-0-133-131 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 22:22:26.005993 ip-10-0-133-131 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 22:22:26.005993 ip-10-0-133-131 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 22:22:26.005993 ip-10-0-133-131 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 22:22:26.005993 ip-10-0-133-131 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 22:22:26.005993 ip-10-0-133-131 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 22:22:26.005993 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.004354    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010001    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010012    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010016    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010019    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010023    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010026    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010029    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010032    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010035    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010042    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010045    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010048    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010051    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010054    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010056    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010059    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010061    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010064    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010066    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 22:22:26.015354 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010069    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010072    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010074    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010077    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010081    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010085    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010088    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010091    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010093    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010096    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010100    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010103    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010106    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010109    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010111    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010114    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010116    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010119    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010121    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 22:22:26.018459 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010124    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010127    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010131    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010134    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010137    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010139    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010142    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010145    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010148    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010150    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010153    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010155    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010158    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010161    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010163    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010165    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010168    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010170    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010173    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010175    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 22:22:26.022604 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010178    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010182    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010186    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010189    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010192    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010195    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010197    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010200    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010202    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010205    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010207    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010210    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010213    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010216    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010219    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010221    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010224    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010226    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010249    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 22:22:26.025418 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010253    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010256    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010259    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010262    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010265    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010268    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010270    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010273    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010276    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010278    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010281    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010283    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010286    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010288    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010291    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010293    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010295    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010298    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010300    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010303    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 22:22:26.028913 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010306    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.010308    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011022    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011029    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011031    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011034    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011037    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011043    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011047    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011051    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011054    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011057    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011060    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011063    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011067    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011070    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011073    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011076    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011080    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011083    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 22:22:26.031790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011086    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011089    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011091    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011094    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011097    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011099    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011101    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011104    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011106    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011110    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011112    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011115    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011117    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011120    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011122    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011125    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011127    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011130    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011134    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 22:22:26.036170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011138    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011142    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011145    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011148    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011150    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011153    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011156    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011158    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011161    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011163    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011166    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011168    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011171    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011173    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011176    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011178    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011180    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011183    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011185    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011190    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 22:22:26.056799 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011192    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011195    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011198    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011200    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011203    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011205    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011208    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011210    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011213    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011215    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011218    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011220    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011223    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011225    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011229    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011231    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011234    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011236    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011238    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011241    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 22:22:26.059236 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011243    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011246    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011249    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011251    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011254    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011257    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011259    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011262    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011265    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011267    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011270    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011272    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011275    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011277    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011280    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011282    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011285    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011288    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011290    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011293    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 22:22:26.073760 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011295    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011298    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011692    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011697    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011700    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011703    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011707    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011709    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011712    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011716    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011720    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011722    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011725    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011728    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011733    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011736    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011739    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011741    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011744    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011746    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 22:22:26.076385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011749    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011751    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011754    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011756    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011759    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011761    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011764    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011766    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011769    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011771    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011774    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011776    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011779    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011782    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011784    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011787    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011790    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011792    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011795    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 22:22:26.094991 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011798    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 22:22:26.093580 ip-10-0-133-131 systemd[1]: Started Kubernetes Kubelet.
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011801    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011803    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011805    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011808    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011810    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011814    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011819    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011821    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011824    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011827    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011829    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011831    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011836    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011838    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011841    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011844    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011846    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011849    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011851    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 22:22:26.098847 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011854    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011857    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011860    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011862    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011864    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011867    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011870    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011872    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011875    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011877    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011880    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011882    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011885    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011888    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011890    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011893    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011896    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011898    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011900    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011905    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 22:22:26.297611 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011907    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011910    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011913    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011915    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011934    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011940    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011943    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011946    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011949    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011951    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011954    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011957    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011960    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011963    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011965    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011968    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011971    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011973    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011976    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 22:22:26.318341 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011978    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011981    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.011983    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012051    2583 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012059    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012066    2583 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012073    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012092    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012096    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012100    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012104    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012107    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012111    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012115    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012118    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012121    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012124    2583 flags.go:64] FLAG: --cgroup-root=""
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012126    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012131    2583 flags.go:64] FLAG: --client-ca-file=""
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012133    2583 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012136    2583 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012143    2583 flags.go:64] FLAG: --cluster-domain=""
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012146    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012149    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 22:22:26.321495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012152    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012156    2583 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012159    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012162    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012165    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012169    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012172    2583 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012175    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012178    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012181    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012184    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012188    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012191    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012194    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012202    2583 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012205    2583 flags.go:64] FLAG: --enable-server="true"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012214    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012220    2583 flags.go:64] FLAG: --event-burst="100"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012224    2583 flags.go:64] FLAG: --event-qps="50"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012226    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012230    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012233    2583 flags.go:64] FLAG: --eviction-hard=""
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012237    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012239    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012242    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 22:22:26.322730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012245    2583 flags.go:64] FLAG: --eviction-soft=""
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012250    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012253    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012256    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012259    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012261    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012264    2583 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012267    2583 flags.go:64] FLAG: --feature-gates=""
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012271    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012274    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012277    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012280    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012283    2583 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012286    2583 flags.go:64] FLAG: --help="false"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012289    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012292    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012295    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012298    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012301    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012304    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012307    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012310    2583 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012312    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 22:22:26.324996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012315    2583 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012323    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012326    2583 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012329    2583 flags.go:64] FLAG: --kube-reserved=""
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012334    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012337    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012339    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012342    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012354    2583 flags.go:64] FLAG: --lock-file=""
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012357    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012361    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012364    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012370    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012373    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012376    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012378    2583 flags.go:64] FLAG: --logging-format="text"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012381    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012384    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012387    2583 flags.go:64] FLAG: --manifest-url=""
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012390    2583 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012394    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012397    2583 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012401    2583 flags.go:64] FLAG: --max-pods="110"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012404    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012407    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 22:22:26.334261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012409    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012412    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012415    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012418    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012421    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012428    2583 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012431    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012433    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012436    2583 flags.go:64] FLAG: --pod-cidr=""
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012440    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012446    2583 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012449    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012453    2583 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012455    2583 flags.go:64] FLAG: --port="10250"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012458    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012461    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0523fe4ad1c93e0c8"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012464    2583 flags.go:64] FLAG: --qos-reserved=""
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012469    2583 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012472    2583 flags.go:64] FLAG: --register-node="true"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012475    2583 flags.go:64] FLAG: --register-with-taints=""
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012478    2583 flags.go:64] FLAG: --registry-burst="10"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012481    2583 flags.go:64] FLAG: --registry-qps="5"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012484    2583 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012486    2583 flags.go:64] FLAG: --reserved-memory=""
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012490    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 22:22:26.335437 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012493    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012496    2583 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012498    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012501    2583 flags.go:64] FLAG: --runonce="false"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012504    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012507    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012509    2583 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012512    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012515    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012518    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012521    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012524    2583 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012527    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012530    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012533    2583 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012535    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012538    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012541    2583 flags.go:64] FLAG: --system-cgroups=""
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012544    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012550    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012553    2583 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012556    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012560    2583 flags.go:64] FLAG: --tls-min-version=""
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012562    2583 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012566    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 22:22:26.336672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012569    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012572    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012575    2583 flags.go:64] FLAG: --v="2"
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012578    2583 flags.go:64] FLAG: --version="false"
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012582    2583 flags.go:64] FLAG: --vmodule=""
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012586    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.012589    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012717    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012720    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012723    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012726    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012729    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012732    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012735    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012738    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012740    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012743    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012746    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012748    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012752    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012756    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 22:22:26.338660 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012759    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012761    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012764    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012767    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012770    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012775    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012777    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012780    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012783    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012785    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012789    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012792    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012794    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012797    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012799    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012802    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012804    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012807    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012809    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012812    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 22:22:26.339465 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012814    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012817    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012819    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012822    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012824    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012827    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012830    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012832    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012835    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012837    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012839    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012842    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012844    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012847    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012849    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012852    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012854    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012859    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012861    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 22:22:26.340294 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012864    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012866    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012869    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012874    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012878    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012881    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012884    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012886    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012889    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012893    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012897    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012900    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012902    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012904    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012907    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012909    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012912    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012914    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012917    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 22:22:26.343517 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012936    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012939    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012942    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012944    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012947    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012949    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012952    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012955    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012957    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012960    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012963    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012967    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012970    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012973    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012975    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012978    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012982    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012984    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012987    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012989    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 22:22:26.344272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012992    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012994    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012997    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.012999    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.013002    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.013005    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.013007    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.013764    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.023991    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.024006    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.024026    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.024031    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024085    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024090    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024093    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 22:22:26.346320 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024096    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024099    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024102    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024105    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024110    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024114    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024117    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024120    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024123    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024126    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024129    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024131    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024134    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024137    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024139    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024142    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024145    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024148    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 22:22:26.347111 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024152    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024154    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024158    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024161    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024163    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024166    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024169    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024172    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024175    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024178    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024181    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024183    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024186    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024189    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024191    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024194    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024196    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024199    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024202    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024207    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024210    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 22:22:26.349130 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024212    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024215    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024218    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024220    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024223    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024226    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024229    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024231    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024234    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024236    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024239    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024242    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024244    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024247    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024250    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024252    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024255    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024258    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024260    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024263    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 22:22:26.349910 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024266    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024269    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024271    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024274    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024276    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024279    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024282    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024285    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024287    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024290    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024294    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024297    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024301    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024304    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024306    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024309    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024311    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024314    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024316    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 22:22:26.350748 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024319    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024322    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024324    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024327    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024329    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024332    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024335    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024337    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024340    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024342    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024357    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024360    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024362    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024365    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024368    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024371    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024374    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 22:22:26.351541 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024376    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.024381    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024493    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024499    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024502    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024504    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024509    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024511    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024514    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024517    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024519    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024522    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024524    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024527    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024529    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 22:22:26.352385 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024532    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024534    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024538    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024541    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024545    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024547    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024550    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024553    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024556    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024559    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024561    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024566    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024569    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024572    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024575    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024578    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024581    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024584    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 22:22:26.353171 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024586    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024589    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024593    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024596    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024600    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024604    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024606    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024609    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024611    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024614    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024617    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024620    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024622    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024625    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024627    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024630    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024633    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024635    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024638    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 22:22:26.353985 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024640    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024643    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024646    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024648    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024651    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024653    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024656    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024658    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024661    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024663    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024666    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024669    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024671    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024674    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024676    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024680    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024683    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024686    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024690    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024693    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 22:22:26.354771 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024695    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024698    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024701    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024703    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024706    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024710    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024713    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024716    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024719    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024722    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024725    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024727    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024729    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024732    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024734    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024737    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024740    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024742    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024744    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024747    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 22:22:26.355596 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024749    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024752    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024754    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024757    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024759    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024762    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024764    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024768    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:26.024771    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.024776    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.024880    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.027251    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.029084    2583 server.go:1019] "Starting client certificate rotation"
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.029183    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 22:22:26.356364 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.029942    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 22:22:26.356759 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.036901    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 22:22:26.356759 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.038949    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 22:22:26.356759 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.049768    2583 log.go:25] "Validated CRI v1 runtime API"
Jul 23 22:22:26.356759 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.052224    2583 log.go:25] "Validated CRI v1 image API"
Jul 23 22:22:26.356759 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.053945    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 22:22:26.356759 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.056494    2583 fs.go:135] Filesystem UUIDs: map[4609d4f8-176e-4cec-bc49-fae004b99f60:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fe543eb5-9ebc-4153-a60f-5c055fbeab4c:/dev/nvme0n1p4]
Jul 23 22:22:26.356759 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.056509    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.061856    2583 manager.go:216] Machine: {Timestamp:2026-07-23 22:22:26.060381311 +0000 UTC m=+0.377910112 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095032 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec227d90efb8e84c9e2de4fa51fcb745 SystemUUID:ec227d90-efb8-e84c-9e2d-e4fa51fcb745 BootID:716276d5-41cd-483b-8953-8be8b950f0c0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ba:fe:bf:fc:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ba:fe:bf:fc:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:74:17:d7:19:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.061973    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.062048    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.064366    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.065512    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.065541    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-131.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.067151    2583 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.067166    2583 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.067177    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.067936    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.069884    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.074700    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.074820    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.074854    2583 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.074872    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.074887    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 22:22:26.455804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.074896    2583 kubelet.go:398] "Adding apiserver pod source"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.074904    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.075259    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.075273    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.077627    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.080034    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.080062    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087068    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087304    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087316    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087327    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087339    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087373    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087385    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087393    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087422    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087445    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087458    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.087466    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.088411    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.088431    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.092668    2583 server.go:1268] "Started kubelet"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.092765    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.092794    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.092876    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.093106    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.093173    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.093325    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-131.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.095027    2583 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.095045    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.098666    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.101085    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-131.ec2.internal.18c50b62c0dc663b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-131.ec2.internal,UID:ip-10-0-133-131.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-131.ec2.internal,},FirstTimestamp:2026-07-23 22:22:26.092639803 +0000 UTC m=+0.410168604,LastTimestamp:2026-07-23 22:22:26.092639803 +0000 UTC m=+0.410168604,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-131.ec2.internal,}"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.104834    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.104843    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.104974    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105604    2583 factory.go:55] Registering systemd factory
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105618    2583 factory.go:223] Registration of the systemd container factory successfully
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105705    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105708    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105729    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105812    2583 factory.go:153] Registering CRI-O factory
Jul 23 22:22:26.456507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105826    2583 factory.go:223] Registration of the crio container factory successfully
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105885    2583 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105894    2583 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105896    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105944    2583 factory.go:103] Registering Raw factory
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.105961    2583 manager.go:1196] Started watching for new ooms in manager
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.106382    2583 manager.go:349] Starting recovery of all containers
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.114981    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.115083    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.115196    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dw9sh"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.127733    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dw9sh"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.128118    2583 manager.go:354] Recovery completed
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.132730    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.134553    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.134584    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.134599    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.135178    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.135190    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.135192    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.135203    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.135918    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.135955    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.135965    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.137780    2583 policy_none.go:49] "None policy: Start"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.137794    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.137804    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.139798    2583 policy_none.go:47] "Start"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.179548    2583 manager.go:337] "Starting Device Plugin manager"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.179582    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.179597    2583 server.go:87] "Starting device plugin registration server"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.179858    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.179872    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.179962    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.180041    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.180050    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.180537    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.180587    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.221302    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.222467    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.222502    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.222524    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.222561    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.225038    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.280495    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.281524    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.281551    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 22:22:26.459335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.281565    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.281594    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.287971    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.287988    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-131.ec2.internal\": node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.318120    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.323207    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"]
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.323297    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.324234    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.324265    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.324279    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.327027    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.327126    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.327162    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.327744    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.327770    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.327745    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.327785    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.327805    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.327816    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.330411    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.330437    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.331643    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.331668    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.331681    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.408249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fa81d99861b4ec4b271cd1e93427d67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"3fa81d99861b4ec4b271cd1e93427d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.408275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa81d99861b4ec4b271cd1e93427d67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"3fa81d99861b4ec4b271cd1e93427d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.408291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63161ea256ddec4bd72e56dccd7997d2-config\") pod \"kube-apiserver-proxy-ip-10-0-133-131.ec2.internal\" (UID: \"63161ea256ddec4bd72e56dccd7997d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.460909 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.418335    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:26.572785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.508732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fa81d99861b4ec4b271cd1e93427d67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"3fa81d99861b4ec4b271cd1e93427d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.572785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.508769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa81d99861b4ec4b271cd1e93427d67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"3fa81d99861b4ec4b271cd1e93427d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.572785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.508786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63161ea256ddec4bd72e56dccd7997d2-config\") pod \"kube-apiserver-proxy-ip-10-0-133-131.ec2.internal\" (UID: \"63161ea256ddec4bd72e56dccd7997d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.572785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.508821    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fa81d99861b4ec4b271cd1e93427d67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"3fa81d99861b4ec4b271cd1e93427d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.572785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.508818    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63161ea256ddec4bd72e56dccd7997d2-config\") pod \"kube-apiserver-proxy-ip-10-0-133-131.ec2.internal\" (UID: \"63161ea256ddec4bd72e56dccd7997d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.572785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.508843    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa81d99861b4ec4b271cd1e93427d67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"3fa81d99861b4ec4b271cd1e93427d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.572785 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.518832    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:26.694954 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.619492    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:26.694954 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.694727    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-131.ec2.internal\" not found" node="ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.697700 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.697481    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.842177 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.699340    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-131.ec2.internal\" not found" node="ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.842177 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.701446    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:26.842177 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.719846    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:26.842177 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.820306    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:26.973488 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:26.920956    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:26.973488 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:26.921499    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 22:22:27.104679 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:27.021208    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:27.104679 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.028390    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 22:22:27.104679 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.028537    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 22:22:27.104679 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.028580    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 22:22:27.122497 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:27.121629    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:27.233502 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.133894    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 22:17:26 +0000 UTC" deadline="2028-04-14 10:52:11.741268858 +0000 UTC"
Jul 23 22:22:27.233502 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.133947    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15132h29m44.607326165s"
Jul 23 22:22:27.233502 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:27.221783    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:27.352562 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:27.322511    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Jul 23 22:22:27.352562 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.340019    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 22:22:27.359246 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.357188    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 22:22:27.405546 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.405524    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:27.424666 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.424645    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 22:22:27.425472 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.425455    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Jul 23 22:22:27.435689 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.435672    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 22:22:27.614055 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:27.613979    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63161ea256ddec4bd72e56dccd7997d2.slice/crio-4dcf26c1076e62f36df4a6d2afa4beec9111afb08e62aac0653612e33dddebe7 WatchSource:0}: Error finding container 4dcf26c1076e62f36df4a6d2afa4beec9111afb08e62aac0653612e33dddebe7: Status 404 returned error can't find the container with id 4dcf26c1076e62f36df4a6d2afa4beec9111afb08e62aac0653612e33dddebe7
Jul 23 22:22:27.620831 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:27.620810    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 22:22:27.623968 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:27.623924    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa81d99861b4ec4b271cd1e93427d67.slice/crio-7864a41f772b51767c64b01ae2163185116fcbb500de4e9fabb31bf6923b40cb WatchSource:0}: Error finding container 7864a41f772b51767c64b01ae2163185116fcbb500de4e9fabb31bf6923b40cb: Status 404 returned error can't find the container with id 7864a41f772b51767c64b01ae2163185116fcbb500de4e9fabb31bf6923b40cb
Jul 23 22:22:28.076809 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.076732    2583 apiserver.go:52] "Watching apiserver"
Jul 23 22:22:28.087645 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.087614    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 22:22:28.088586 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.088563    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 22:22:28.088950 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.088927    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-92v68","kube-system/global-pull-secret-syncer-mtsxj","kube-system/konnectivity-agent-kwlkg","kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal","openshift-cluster-node-tuning-operator/tuned-t6h69","openshift-image-registry/node-ca-4xjxj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal","openshift-multus/multus-additional-cni-plugins-w5kzg","openshift-multus/network-metrics-daemon-d6b9c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj","openshift-dns/node-resolver-68765","openshift-multus/multus-2bk9g","openshift-network-diagnostics/network-check-target-tvd42","openshift-network-operator/iptables-alerter-ss95p"]
Jul 23 22:22:28.091644 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.091623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:28.091747 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.091702    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:28.094709 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.094692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:28.094805 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.094752    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:28.097396 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.097373    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:28.099938 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.099893    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.100476 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.100245    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 22:22:28.100476 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.100361    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sbtvf\""
Jul 23 22:22:28.100476 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.100422    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 22:22:28.102409 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.102390    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 22:22:28.102552 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.102511    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 22:22:28.102733 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.102710    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 22:22:28.102843 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.102797    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 22:22:28.102972 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.102936    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tkkw9\""
Jul 23 22:22:28.103141 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.103118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.103341 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.103324    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 22:22:28.103462 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.103425    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 22:22:28.105043 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.104981    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xjxj"
Jul 23 22:22:28.105666 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.105228    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 22:22:28.105666 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.105536    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 22:22:28.105666 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.105604    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t47xv\""
Jul 23 22:22:28.105817 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.105679    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 22:22:28.107344 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.107324    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 22:22:28.107651 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.107610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ss95p"
Jul 23 22:22:28.107740 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.107677    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 22:22:28.108042 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.107815    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 22:22:28.108208 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.108177    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9j4g7\""
Jul 23 22:22:28.110160 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.110137    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 22:22:28.110281 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.110263    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 22:22:28.110349 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.110272    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d6tlr\""
Jul 23 22:22:28.110578 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.110558    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 22:22:28.110700 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.110677    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.114850 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.114720    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 22:22:28.114850 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.114768    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdkcq\""
Jul 23 22:22:28.114850 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.114796    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 22:22:28.115062 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.114895    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 22:22:28.115243 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.115220    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 22:22:28.115461 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.115430    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 22:22:28.116628 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.116604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:28.116746 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.116711    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:28.117273 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/889cb48e-740e-4093-9908-67f62a225840-serviceca\") pod \"node-ca-4xjxj\" (UID: \"889cb48e-740e-4093-9908-67f62a225840\") " pod="openshift-image-registry/node-ca-4xjxj"
Jul 23 22:22:28.117377 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117290    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-systemd-units\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.117377 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-run-systemd\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.117377 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-env-overrides\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.117546 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-ovnkube-script-lib\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.117546 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-lib-modules\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.117546 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-var-lib-kubelet\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.117546 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5nwv\" (UniqueName: \"kubernetes.io/projected/b82bbc6a-0156-4fa4-846f-3302c983648e-kube-api-access-n5nwv\") pod \"iptables-alerter-ss95p\" (UID: \"b82bbc6a-0156-4fa4-846f-3302c983648e\") " pod="openshift-network-operator/iptables-alerter-ss95p"
Jul 23 22:22:28.117546 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.117818 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68c30bcd-6fb8-4d32-adcc-881a10cdc158-konnectivity-ca\") pod \"konnectivity-agent-kwlkg\" (UID: \"68c30bcd-6fb8-4d32-adcc-881a10cdc158\") " pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:28.117818 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsbn8\" (UniqueName: \"kubernetes.io/projected/889cb48e-740e-4093-9908-67f62a225840-kube-api-access-qsbn8\") pod \"node-ca-4xjxj\" (UID: \"889cb48e-740e-4093-9908-67f62a225840\") " pod="openshift-image-registry/node-ca-4xjxj"
Jul 23 22:22:28.117818 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/729340b2-7e6e-474a-8475-db80f5a1ac8b-kubelet-config\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:28.117818 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-kubelet\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.117818 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117702    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-node-log\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.117818 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-tuned\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.117818 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b82bbc6a-0156-4fa4-846f-3302c983648e-iptables-alerter-script\") pod \"iptables-alerter-ss95p\" (UID: \"b82bbc6a-0156-4fa4-846f-3302c983648e\") " pod="openshift-network-operator/iptables-alerter-ss95p"
Jul 23 22:22:28.117818 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-etc-openvswitch\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.117818 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckvbj\" (UniqueName: \"kubernetes.io/projected/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-kube-api-access-ckvbj\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-systemd\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-run\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117906    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-host\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mrg\" (UniqueName: \"kubernetes.io/projected/7e4891f9-35db-4acd-9b5f-917a5d8917db-kube-api-access-l9mrg\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.117978    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68c30bcd-6fb8-4d32-adcc-881a10cdc158-agent-certs\") pod \"konnectivity-agent-kwlkg\" (UID: \"68c30bcd-6fb8-4d32-adcc-881a10cdc158\") " pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118038    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-run-ovn\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118078    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-cni-bin\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-kubernetes\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-sys\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b82bbc6a-0156-4fa4-846f-3302c983648e-host-slash\") pod \"iptables-alerter-ss95p\" (UID: \"b82bbc6a-0156-4fa4-846f-3302c983648e\") " pod="openshift-network-operator/iptables-alerter-ss95p"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/889cb48e-740e-4093-9908-67f62a225840-host\") pod \"node-ca-4xjxj\" (UID: \"889cb48e-740e-4093-9908-67f62a225840\") " pod="openshift-image-registry/node-ca-4xjxj"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-run-openvswitch\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-cni-netd\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.118296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-ovnkube-config\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6lg5\" (UniqueName: \"kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5\") pod \"network-check-target-tvd42\" (UID: \"492cf2fb-60f4-494f-8608-72d8f00b071f\") " pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/729340b2-7e6e-474a-8475-db80f5a1ac8b-dbus\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-var-lib-openvswitch\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-ovn-node-metrics-cert\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118366    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-sysconfig\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-sysctl-d\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118416    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e4891f9-35db-4acd-9b5f-917a5d8917db-tmp\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-run-netns\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118483    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-slash\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-log-socket\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118550    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-modprobe-d\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.119135 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.118594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-sysctl-conf\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.120520 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.120475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.123304 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.123037    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 22:22:28.123304 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.123038    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-49drd\""
Jul 23 22:22:28.123304 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.123219    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 22:22:28.123304 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.123227    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 22:22:28.123564 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.123329    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-68765"
Jul 23 22:22:28.126023 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.125987    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 22:22:28.126463 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.126438    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.126575 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.126443    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 22:22:28.126846 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.126823    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pqsdz\""
Jul 23 22:22:28.128761 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.128742    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 22:22:28.128870 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.128829    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g4xrb\""
Jul 23 22:22:28.139825 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.139806    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-trh7g"
Jul 23 22:22:28.147474 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.147456    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-trh7g"
Jul 23 22:22:28.207362 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.207302    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.219715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-systemd\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.219761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-system-cni-dir\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.219789    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-cnibin\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.219816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zph9\" (UniqueName: \"kubernetes.io/projected/cd941d5f-38b3-49b4-a92e-afcec5348fb2-kube-api-access-4zph9\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.219843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-system-cni-dir\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.219866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-run-multus-certs\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.219892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-cni-bin\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.219919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b82bbc6a-0156-4fa4-846f-3302c983648e-host-slash\") pod \"iptables-alerter-ss95p\" (UID: \"b82bbc6a-0156-4fa4-846f-3302c983648e\") " pod="openshift-network-operator/iptables-alerter-ss95p"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.219960    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd941d5f-38b3-49b4-a92e-afcec5348fb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.219986    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-cni-dir\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/889cb48e-740e-4093-9908-67f62a225840-host\") pod \"node-ca-4xjxj\" (UID: \"889cb48e-740e-4093-9908-67f62a225840\") " pod="openshift-image-registry/node-ca-4xjxj"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-run-openvswitch\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-cni-netd\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.221312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-os-release\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd941d5f-38b3-49b4-a92e-afcec5348fb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/729340b2-7e6e-474a-8475-db80f5a1ac8b-dbus\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-ovn-node-metrics-cert\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89ffe12b-74bf-4582-8395-b47766839015-hosts-file\") pod \"node-resolver-68765\" (UID: \"89ffe12b-74bf-4582-8395-b47766839015\") " pod="openshift-dns/node-resolver-68765"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp6qh\" (UniqueName: \"kubernetes.io/projected/051cb605-0708-4f32-bf6e-1e687d328a0a-kube-api-access-kp6qh\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-run-netns\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-log-socket\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-modprobe-d\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220371    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-socket-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220403    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-registration-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220429    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4z8f\" (UniqueName: \"kubernetes.io/projected/012cefdf-5d65-4f60-aa38-35e098c7f8be-kube-api-access-v4z8f\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-systemd-units\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-run-systemd\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-var-lib-kubelet\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hkpw\" (UniqueName: \"kubernetes.io/projected/89ffe12b-74bf-4582-8395-b47766839015-kube-api-access-2hkpw\") pod \"node-resolver-68765\" (UID: \"89ffe12b-74bf-4582-8395-b47766839015\") " pod="openshift-dns/node-resolver-68765"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-var-lib-kubelet\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.222291 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68c30bcd-6fb8-4d32-adcc-881a10cdc158-konnectivity-ca\") pod \"konnectivity-agent-kwlkg\" (UID: \"68c30bcd-6fb8-4d32-adcc-881a10cdc158\") " pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-kubelet\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220692    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-tuned\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220720    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b82bbc6a-0156-4fa4-846f-3302c983648e-iptables-alerter-script\") pod \"iptables-alerter-ss95p\" (UID: \"b82bbc6a-0156-4fa4-846f-3302c983648e\") " pod="openshift-network-operator/iptables-alerter-ss95p"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-run-k8s-cni-cncf-io\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-etc-kubernetes\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-etc-openvswitch\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckvbj\" (UniqueName: \"kubernetes.io/projected/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-kube-api-access-ckvbj\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-run\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-host\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mrg\" (UniqueName: \"kubernetes.io/projected/7e4891f9-35db-4acd-9b5f-917a5d8917db-kube-api-access-l9mrg\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd941d5f-38b3-49b4-a92e-afcec5348fb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220979    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-systemd\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.220987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-etc-selinux\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221059    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-hostroot\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221159    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-systemd-units\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.223188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221198    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-modprobe-d\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221208    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-run-systemd\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221268    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-var-lib-kubelet\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-kubelet\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221394    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/889cb48e-740e-4093-9908-67f62a225840-host\") pod \"node-ca-4xjxj\" (UID: \"889cb48e-740e-4093-9908-67f62a225840\") " pod="openshift-image-registry/node-ca-4xjxj"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221442    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-run-openvswitch\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-cni-netd\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68c30bcd-6fb8-4d32-adcc-881a10cdc158-agent-certs\") pod \"konnectivity-agent-kwlkg\" (UID: \"68c30bcd-6fb8-4d32-adcc-881a10cdc158\") " pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221736    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221686    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221814    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/729340b2-7e6e-474a-8475-db80f5a1ac8b-dbus\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-run-ovn-kubernetes\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.221930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68c30bcd-6fb8-4d32-adcc-881a10cdc158-konnectivity-ca\") pod \"konnectivity-agent-kwlkg\" (UID: \"68c30bcd-6fb8-4d32-adcc-881a10cdc158\") " pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-cni-bin\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.222200    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.222419    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret podName:729340b2-7e6e-474a-8475-db80f5a1ac8b nodeName:}" failed. No retries permitted until 2026-07-23 22:22:28.722384276 +0000 UTC m=+3.039913078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret") pod "global-pull-secret-syncer-mtsxj" (UID: "729340b2-7e6e-474a-8475-db80f5a1ac8b") : object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-run-ovn\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.224184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-host\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-kubernetes\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222531    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b82bbc6a-0156-4fa4-846f-3302c983648e-iptables-alerter-script\") pod \"iptables-alerter-ss95p\" (UID: \"b82bbc6a-0156-4fa4-846f-3302c983648e\") " pod="openshift-network-operator/iptables-alerter-ss95p"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-sys\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkcj5\" (UniqueName: \"kubernetes.io/projected/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-kube-api-access-hkcj5\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-var-lib-cni-bin\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-ovnkube-config\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222751    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-conf-dir\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b82bbc6a-0156-4fa4-846f-3302c983648e-host-slash\") pod \"iptables-alerter-ss95p\" (UID: \"b82bbc6a-0156-4fa4-846f-3302c983648e\") " pod="openshift-network-operator/iptables-alerter-ss95p"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222770    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lg5\" (UniqueName: \"kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5\") pod \"network-check-target-tvd42\" (UID: \"492cf2fb-60f4-494f-8608-72d8f00b071f\") " pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-var-lib-openvswitch\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.222857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-run\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-sysconfig\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-sysctl-d\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e4891f9-35db-4acd-9b5f-917a5d8917db-tmp\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-socket-dir-parent\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-run-netns\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.225262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-sys-fs\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-os-release\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-daemon-config\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-run-ovn\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-slash\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-kubernetes\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-sysctl-conf\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/051cb605-0708-4f32-bf6e-1e687d328a0a-cni-binary-copy\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/889cb48e-740e-4093-9908-67f62a225840-serviceca\") pod \"node-ca-4xjxj\" (UID: \"889cb48e-740e-4093-9908-67f62a225840\") " pod="openshift-image-registry/node-ca-4xjxj"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-env-overrides\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-etc-openvswitch\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223684    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-ovnkube-script-lib\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223501    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-ovnkube-config\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.223714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-lib-modules\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.224081    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-sys\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.224312    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-slash\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.224387    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-sysconfig\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.226199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.224450    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-sysctl-conf\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.227128 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.224554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/889cb48e-740e-4093-9908-67f62a225840-serviceca\") pod \"node-ca-4xjxj\" (UID: \"889cb48e-740e-4093-9908-67f62a225840\") " pod="openshift-image-registry/node-ca-4xjxj"
Jul 23 22:22:28.227128 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.224563    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-host-run-netns\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.227128 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.224606    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-log-socket\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.227128 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.224698    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-sysctl-d\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.227128 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.224768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e4891f9-35db-4acd-9b5f-917a5d8917db-lib-modules\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.227128 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.224323    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-var-lib-openvswitch\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.230285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.230258    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-ovn-node-metrics-cert\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.230395 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.230339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-ovnkube-script-lib\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.230395 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.230353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-env-overrides\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.230514 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.230421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5nwv\" (UniqueName: \"kubernetes.io/projected/b82bbc6a-0156-4fa4-846f-3302c983648e-kube-api-access-n5nwv\") pod \"iptables-alerter-ss95p\" (UID: \"b82bbc6a-0156-4fa4-846f-3302c983648e\") " pod="openshift-network-operator/iptables-alerter-ss95p"
Jul 23 22:22:28.230514 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.230456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e4891f9-35db-4acd-9b5f-917a5d8917db-etc-tuned\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.230625 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.230539    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-device-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.230800 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.230774    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ffe12b-74bf-4582-8395-b47766839015-tmp-dir\") pod \"node-resolver-68765\" (UID: \"89ffe12b-74bf-4582-8395-b47766839015\") " pod="openshift-dns/node-resolver-68765"
Jul 23 22:22:28.230875 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.230814    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68c30bcd-6fb8-4d32-adcc-881a10cdc158-agent-certs\") pod \"konnectivity-agent-kwlkg\" (UID: \"68c30bcd-6fb8-4d32-adcc-881a10cdc158\") " pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:28.230875 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.230834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-cnibin\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.230989 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.230911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsbn8\" (UniqueName: \"kubernetes.io/projected/889cb48e-740e-4093-9908-67f62a225840-kube-api-access-qsbn8\") pod \"node-ca-4xjxj\" (UID: \"889cb48e-740e-4093-9908-67f62a225840\") " pod="openshift-image-registry/node-ca-4xjxj"
Jul 23 22:22:28.231143 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.231118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/729340b2-7e6e-474a-8475-db80f5a1ac8b-kubelet-config\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:28.231210 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.231169    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 22:22:28.231210 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.231206    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 22:22:28.231306 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.231221    2583 projected.go:196] Error preparing data for projected volume kube-api-access-x6lg5 for pod openshift-network-diagnostics/network-check-target-tvd42: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:28.231306 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.231236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-node-log\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.231400 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.231314    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/729340b2-7e6e-474a-8475-db80f5a1ac8b-kubelet-config\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:28.231453 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.231400    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5 podName:492cf2fb-60f4-494f-8608-72d8f00b071f nodeName:}" failed. No retries permitted until 2026-07-23 22:22:28.73137275 +0000 UTC m=+3.048901557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x6lg5" (UniqueName: "kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5") pod "network-check-target-tvd42" (UID: "492cf2fb-60f4-494f-8608-72d8f00b071f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:28.231703 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.231597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:28.231703 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.231637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-var-lib-cni-multus\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.231860 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.231809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e4891f9-35db-4acd-9b5f-917a5d8917db-tmp\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.232231 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.231812    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-node-log\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.234902 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.234453    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mrg\" (UniqueName: \"kubernetes.io/projected/7e4891f9-35db-4acd-9b5f-917a5d8917db-kube-api-access-l9mrg\") pod \"tuned-t6h69\" (UID: \"7e4891f9-35db-4acd-9b5f-917a5d8917db\") " pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.237971 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.237946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckvbj\" (UniqueName: \"kubernetes.io/projected/fcd3c0d0-f7ad-483b-bb05-c55094232c7a-kube-api-access-ckvbj\") pod \"ovnkube-node-92v68\" (UID: \"fcd3c0d0-f7ad-483b-bb05-c55094232c7a\") " pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.242091 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.242069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5nwv\" (UniqueName: \"kubernetes.io/projected/b82bbc6a-0156-4fa4-846f-3302c983648e-kube-api-access-n5nwv\") pod \"iptables-alerter-ss95p\" (UID: \"b82bbc6a-0156-4fa4-846f-3302c983648e\") " pod="openshift-network-operator/iptables-alerter-ss95p"
Jul 23 22:22:28.244902 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.244859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsbn8\" (UniqueName: \"kubernetes.io/projected/889cb48e-740e-4093-9908-67f62a225840-kube-api-access-qsbn8\") pod \"node-ca-4xjxj\" (UID: \"889cb48e-740e-4093-9908-67f62a225840\") " pod="openshift-image-registry/node-ca-4xjxj"
Jul 23 22:22:28.255901 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.255851    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal" event={"ID":"63161ea256ddec4bd72e56dccd7997d2","Type":"ContainerStarted","Data":"4dcf26c1076e62f36df4a6d2afa4beec9111afb08e62aac0653612e33dddebe7"}
Jul 23 22:22:28.258392 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.258366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" event={"ID":"3fa81d99861b4ec4b271cd1e93427d67","Type":"ContainerStarted","Data":"7864a41f772b51767c64b01ae2163185116fcbb500de4e9fabb31bf6923b40cb"}
Jul 23 22:22:28.333173 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333088    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcj5\" (UniqueName: \"kubernetes.io/projected/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-kube-api-access-hkcj5\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:28.333173 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-var-lib-cni-bin\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333387 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-conf-dir\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333387 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333211    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-var-lib-cni-bin\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333387 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-socket-dir-parent\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333387 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-sys-fs\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.333387 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333279    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-socket-dir-parent\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333387 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-os-release\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333387 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333324    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-conf-dir\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333387 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-daemon-config\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333387 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.333387 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/051cb605-0708-4f32-bf6e-1e687d328a0a-cni-binary-copy\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-device-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-sys-fs\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ffe12b-74bf-4582-8395-b47766839015-tmp-dir\") pod \"node-resolver-68765\" (UID: \"89ffe12b-74bf-4582-8395-b47766839015\") " pod="openshift-dns/node-resolver-68765"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-cnibin\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-var-lib-cni-multus\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-system-cni-dir\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333616    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-cnibin\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zph9\" (UniqueName: \"kubernetes.io/projected/cd941d5f-38b3-49b4-a92e-afcec5348fb2-kube-api-access-4zph9\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-system-cni-dir\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-run-multus-certs\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd941d5f-38b3-49b4-a92e-afcec5348fb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-cni-dir\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333740    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-os-release\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd941d5f-38b3-49b4-a92e-afcec5348fb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.333968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ffe12b-74bf-4582-8395-b47766839015-tmp-dir\") pod \"node-resolver-68765\" (UID: \"89ffe12b-74bf-4582-8395-b47766839015\") " pod="openshift-dns/node-resolver-68765"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333805    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-var-lib-cni-multus\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-system-cni-dir\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333836    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89ffe12b-74bf-4582-8395-b47766839015-hosts-file\") pod \"node-resolver-68765\" (UID: \"89ffe12b-74bf-4582-8395-b47766839015\") " pod="openshift-dns/node-resolver-68765"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333867    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-cnibin\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333889    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-cnibin\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-daemon-config\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.333974    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.334046    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs podName:6d068205-8a0a-4ae6-b8a2-f831bf0e9752 nodeName:}" failed. No retries permitted until 2026-07-23 22:22:28.834026875 +0000 UTC m=+3.151555667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs") pod "network-metrics-daemon-d6b9c" (UID: "6d068205-8a0a-4ae6-b8a2-f831bf0e9752") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-os-release\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-system-cni-dir\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334123    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-device-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-run-multus-certs\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.333774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89ffe12b-74bf-4582-8395-b47766839015-hosts-file\") pod \"node-resolver-68765\" (UID: \"89ffe12b-74bf-4582-8395-b47766839015\") " pod="openshift-dns/node-resolver-68765"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6qh\" (UniqueName: \"kubernetes.io/projected/051cb605-0708-4f32-bf6e-1e687d328a0a-kube-api-access-kp6qh\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334249    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-run-netns\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.334883 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-socket-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-registration-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4z8f\" (UniqueName: \"kubernetes.io/projected/012cefdf-5d65-4f60-aa38-35e098c7f8be-kube-api-access-v4z8f\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334399    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkpw\" (UniqueName: \"kubernetes.io/projected/89ffe12b-74bf-4582-8395-b47766839015-kube-api-access-2hkpw\") pod \"node-resolver-68765\" (UID: \"89ffe12b-74bf-4582-8395-b47766839015\") " pod="openshift-dns/node-resolver-68765"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334420    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-multus-cni-dir\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-var-lib-kubelet\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd941d5f-38b3-49b4-a92e-afcec5348fb2-os-release\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334460    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-var-lib-kubelet\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-run-netns\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-run-k8s-cni-cncf-io\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-etc-kubernetes\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334515    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-host-run-k8s-cni-cncf-io\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd941d5f-38b3-49b4-a92e-afcec5348fb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334565    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-registration-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-socket-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-etc-selinux\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.335561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-etc-kubernetes\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.336406 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd941d5f-38b3-49b4-a92e-afcec5348fb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.336406 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-hostroot\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.336406 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334682    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/012cefdf-5d65-4f60-aa38-35e098c7f8be-etc-selinux\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.336406 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334690    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/051cb605-0708-4f32-bf6e-1e687d328a0a-hostroot\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.336406 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/051cb605-0708-4f32-bf6e-1e687d328a0a-cni-binary-copy\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.336406 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.334946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd941d5f-38b3-49b4-a92e-afcec5348fb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.336406 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.335116    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd941d5f-38b3-49b4-a92e-afcec5348fb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.346293 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.346241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zph9\" (UniqueName: \"kubernetes.io/projected/cd941d5f-38b3-49b4-a92e-afcec5348fb2-kube-api-access-4zph9\") pod \"multus-additional-cni-plugins-w5kzg\" (UID: \"cd941d5f-38b3-49b4-a92e-afcec5348fb2\") " pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.346911 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.346865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcj5\" (UniqueName: \"kubernetes.io/projected/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-kube-api-access-hkcj5\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:28.347365 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.347340    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6qh\" (UniqueName: \"kubernetes.io/projected/051cb605-0708-4f32-bf6e-1e687d328a0a-kube-api-access-kp6qh\") pod \"multus-2bk9g\" (UID: \"051cb605-0708-4f32-bf6e-1e687d328a0a\") " pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.347928 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.347551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4z8f\" (UniqueName: \"kubernetes.io/projected/012cefdf-5d65-4f60-aa38-35e098c7f8be-kube-api-access-v4z8f\") pod \"aws-ebs-csi-driver-node-48szj\" (UID: \"012cefdf-5d65-4f60-aa38-35e098c7f8be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.347928 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.347879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hkpw\" (UniqueName: \"kubernetes.io/projected/89ffe12b-74bf-4582-8395-b47766839015-kube-api-access-2hkpw\") pod \"node-resolver-68765\" (UID: \"89ffe12b-74bf-4582-8395-b47766839015\") " pod="openshift-dns/node-resolver-68765"
Jul 23 22:22:28.410504 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.410472    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:28.421993 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.421916    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:28.425567 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:28.425370    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c30bcd_6fb8_4d32_adcc_881a10cdc158.slice/crio-e88db3f73103303ab9efb2b490bb53a22995d6d8826faf0e7d3db8599e306ebf WatchSource:0}: Error finding container e88db3f73103303ab9efb2b490bb53a22995d6d8826faf0e7d3db8599e306ebf: Status 404 returned error can't find the container with id e88db3f73103303ab9efb2b490bb53a22995d6d8826faf0e7d3db8599e306ebf
Jul 23 22:22:28.431697 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.430866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t6h69"
Jul 23 22:22:28.436222 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:28.436177    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd3c0d0_f7ad_483b_bb05_c55094232c7a.slice/crio-706156c7146ddccaf73b62f26649d912852404557728d7a3e6cb906a192f9346 WatchSource:0}: Error finding container 706156c7146ddccaf73b62f26649d912852404557728d7a3e6cb906a192f9346: Status 404 returned error can't find the container with id 706156c7146ddccaf73b62f26649d912852404557728d7a3e6cb906a192f9346
Jul 23 22:22:28.437279 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.437263    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xjxj"
Jul 23 22:22:28.445317 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:28.445294    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4891f9_35db_4acd_9b5f_917a5d8917db.slice/crio-636a8e06fd3410801c2a99ca639214685708296f791ca00a85e56dd672d6e4a3 WatchSource:0}: Error finding container 636a8e06fd3410801c2a99ca639214685708296f791ca00a85e56dd672d6e4a3: Status 404 returned error can't find the container with id 636a8e06fd3410801c2a99ca639214685708296f791ca00a85e56dd672d6e4a3
Jul 23 22:22:28.446441 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.446217    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ss95p"
Jul 23 22:22:28.448457 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:28.448359    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889cb48e_740e_4093_9908_67f62a225840.slice/crio-6e2bc37288b2e27100cd064f705fec4e867d7e2b1cd2fc2bf35300e3bed905d2 WatchSource:0}: Error finding container 6e2bc37288b2e27100cd064f705fec4e867d7e2b1cd2fc2bf35300e3bed905d2: Status 404 returned error can't find the container with id 6e2bc37288b2e27100cd064f705fec4e867d7e2b1cd2fc2bf35300e3bed905d2
Jul 23 22:22:28.456333 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.456196    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w5kzg"
Jul 23 22:22:28.459308 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:28.459178    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82bbc6a_0156_4fa4_846f_3302c983648e.slice/crio-6d7f48d2ead87d2aeeec7a0ced90cc9af59ec23f2610518d50d65c4468d9c4d5 WatchSource:0}: Error finding container 6d7f48d2ead87d2aeeec7a0ced90cc9af59ec23f2610518d50d65c4468d9c4d5: Status 404 returned error can't find the container with id 6d7f48d2ead87d2aeeec7a0ced90cc9af59ec23f2610518d50d65c4468d9c4d5
Jul 23 22:22:28.464119 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.464090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj"
Jul 23 22:22:28.469301 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:28.469238    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd941d5f_38b3_49b4_a92e_afcec5348fb2.slice/crio-234f23092d1bca51ad19a364d4ee096220cb29fa0e5cd85a163d0c472e363936 WatchSource:0}: Error finding container 234f23092d1bca51ad19a364d4ee096220cb29fa0e5cd85a163d0c472e363936: Status 404 returned error can't find the container with id 234f23092d1bca51ad19a364d4ee096220cb29fa0e5cd85a163d0c472e363936
Jul 23 22:22:28.473245 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.473084    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-68765"
Jul 23 22:22:28.475912 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:28.475888    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012cefdf_5d65_4f60_aa38_35e098c7f8be.slice/crio-466b8415b9ec70951f7955b8bd1796f5766b3574ac43fa419d65dd0368cf7c0c WatchSource:0}: Error finding container 466b8415b9ec70951f7955b8bd1796f5766b3574ac43fa419d65dd0368cf7c0c: Status 404 returned error can't find the container with id 466b8415b9ec70951f7955b8bd1796f5766b3574ac43fa419d65dd0368cf7c0c
Jul 23 22:22:28.479271 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.479184    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2bk9g"
Jul 23 22:22:28.497314 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:22:28.497284    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051cb605_0708_4f32_bf6e_1e687d328a0a.slice/crio-dd25343c6c5922a5fec066cad935b36132d9cebb7485e44d79adc475d3683f97 WatchSource:0}: Error finding container dd25343c6c5922a5fec066cad935b36132d9cebb7485e44d79adc475d3683f97: Status 404 returned error can't find the container with id dd25343c6c5922a5fec066cad935b36132d9cebb7485e44d79adc475d3683f97
Jul 23 22:22:28.737123 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.737024    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:28.737123 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.737079    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lg5\" (UniqueName: \"kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5\") pod \"network-check-target-tvd42\" (UID: \"492cf2fb-60f4-494f-8608-72d8f00b071f\") " pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:28.737346 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.737196    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:28.737346 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.737274    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret podName:729340b2-7e6e-474a-8475-db80f5a1ac8b nodeName:}" failed. No retries permitted until 2026-07-23 22:22:29.737256846 +0000 UTC m=+4.054785633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret") pod "global-pull-secret-syncer-mtsxj" (UID: "729340b2-7e6e-474a-8475-db80f5a1ac8b") : object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:28.737346 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.737209    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 22:22:28.737518 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.737347    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 22:22:28.737518 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.737367    2583 projected.go:196] Error preparing data for projected volume kube-api-access-x6lg5 for pod openshift-network-diagnostics/network-check-target-tvd42: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:28.737518 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.737432    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5 podName:492cf2fb-60f4-494f-8608-72d8f00b071f nodeName:}" failed. No retries permitted until 2026-07-23 22:22:29.737418928 +0000 UTC m=+4.054947715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6lg5" (UniqueName: "kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5") pod "network-check-target-tvd42" (UID: "492cf2fb-60f4-494f-8608-72d8f00b071f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:28.838386 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:28.838346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:28.838571 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.838481    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:28.838571 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:28.838553    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs podName:6d068205-8a0a-4ae6-b8a2-f831bf0e9752 nodeName:}" failed. No retries permitted until 2026-07-23 22:22:29.838525828 +0000 UTC m=+4.156054617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs") pod "network-metrics-daemon-d6b9c" (UID: "6d068205-8a0a-4ae6-b8a2-f831bf0e9752") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:29.151486 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.151342    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 22:17:28 +0000 UTC" deadline="2028-04-04 01:53:35.787580328 +0000 UTC"
Jul 23 22:22:29.151486 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.151386    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14883h31m6.636200488s"
Jul 23 22:22:29.223707 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.223423    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:29.223707 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:29.223586    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:29.262563 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.262479    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ss95p" event={"ID":"b82bbc6a-0156-4fa4-846f-3302c983648e","Type":"ContainerStarted","Data":"6d7f48d2ead87d2aeeec7a0ced90cc9af59ec23f2610518d50d65c4468d9c4d5"}
Jul 23 22:22:29.264343 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.264246    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92v68" event={"ID":"fcd3c0d0-f7ad-483b-bb05-c55094232c7a","Type":"ContainerStarted","Data":"706156c7146ddccaf73b62f26649d912852404557728d7a3e6cb906a192f9346"}
Jul 23 22:22:29.266156 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.265802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kwlkg" event={"ID":"68c30bcd-6fb8-4d32-adcc-881a10cdc158","Type":"ContainerStarted","Data":"e88db3f73103303ab9efb2b490bb53a22995d6d8826faf0e7d3db8599e306ebf"}
Jul 23 22:22:29.267381 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.267310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w5kzg" event={"ID":"cd941d5f-38b3-49b4-a92e-afcec5348fb2","Type":"ContainerStarted","Data":"234f23092d1bca51ad19a364d4ee096220cb29fa0e5cd85a163d0c472e363936"}
Jul 23 22:22:29.268786 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.268747    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xjxj" event={"ID":"889cb48e-740e-4093-9908-67f62a225840","Type":"ContainerStarted","Data":"6e2bc37288b2e27100cd064f705fec4e867d7e2b1cd2fc2bf35300e3bed905d2"}
Jul 23 22:22:29.270245 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.270219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t6h69" event={"ID":"7e4891f9-35db-4acd-9b5f-917a5d8917db","Type":"ContainerStarted","Data":"636a8e06fd3410801c2a99ca639214685708296f791ca00a85e56dd672d6e4a3"}
Jul 23 22:22:29.271933 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.271886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2bk9g" event={"ID":"051cb605-0708-4f32-bf6e-1e687d328a0a","Type":"ContainerStarted","Data":"dd25343c6c5922a5fec066cad935b36132d9cebb7485e44d79adc475d3683f97"}
Jul 23 22:22:29.273820 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.273744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-68765" event={"ID":"89ffe12b-74bf-4582-8395-b47766839015","Type":"ContainerStarted","Data":"2d63509fd6cf3e3520ef1b92a3e9e1943a513e84ceb5c48a4be300f67730b5aa"}
Jul 23 22:22:29.275769 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.275742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj" event={"ID":"012cefdf-5d65-4f60-aa38-35e098c7f8be","Type":"ContainerStarted","Data":"466b8415b9ec70951f7955b8bd1796f5766b3574ac43fa419d65dd0368cf7c0c"}
Jul 23 22:22:29.746380 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.745414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:29.746380 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.745480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lg5\" (UniqueName: \"kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5\") pod \"network-check-target-tvd42\" (UID: \"492cf2fb-60f4-494f-8608-72d8f00b071f\") " pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:29.746380 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:29.745711    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 22:22:29.746380 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:29.745735    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 22:22:29.746380 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:29.745750    2583 projected.go:196] Error preparing data for projected volume kube-api-access-x6lg5 for pod openshift-network-diagnostics/network-check-target-tvd42: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:29.746380 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:29.745817    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5 podName:492cf2fb-60f4-494f-8608-72d8f00b071f nodeName:}" failed. No retries permitted until 2026-07-23 22:22:31.745796766 +0000 UTC m=+6.063325557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6lg5" (UniqueName: "kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5") pod "network-check-target-tvd42" (UID: "492cf2fb-60f4-494f-8608-72d8f00b071f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:29.746380 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:29.746289    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:29.746380 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:29.746343    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret podName:729340b2-7e6e-474a-8475-db80f5a1ac8b nodeName:}" failed. No retries permitted until 2026-07-23 22:22:31.746327078 +0000 UTC m=+6.063855893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret") pod "global-pull-secret-syncer-mtsxj" (UID: "729340b2-7e6e-474a-8475-db80f5a1ac8b") : object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:29.846993 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.846936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:29.847351 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:29.847143    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:29.847351 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:29.847209    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs podName:6d068205-8a0a-4ae6-b8a2-f831bf0e9752 nodeName:}" failed. No retries permitted until 2026-07-23 22:22:31.847190793 +0000 UTC m=+6.164719584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs") pod "network-metrics-daemon-d6b9c" (UID: "6d068205-8a0a-4ae6-b8a2-f831bf0e9752") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:29.971788 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:29.971746    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 22:22:30.152249 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:30.152142    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 22:17:28 +0000 UTC" deadline="2028-04-24 16:19:09.56693916 +0000 UTC"
Jul 23 22:22:30.152249 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:30.152206    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15377h56m39.414738659s"
Jul 23 22:22:30.225957 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:30.223771    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:30.225957 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:30.223901    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:30.225957 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:30.224420    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:30.225957 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:30.224511    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:31.223230 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:31.222727    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:31.223230 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:31.223097    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:31.310760 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:31.310726    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal" event={"ID":"63161ea256ddec4bd72e56dccd7997d2","Type":"ContainerStarted","Data":"58fe5341d4ba25fb8558ffbdf78c313ade752418e4f01efb728743136f1ce449"}
Jul 23 22:22:31.762913 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:31.762873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:31.763168 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:31.762950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lg5\" (UniqueName: \"kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5\") pod \"network-check-target-tvd42\" (UID: \"492cf2fb-60f4-494f-8608-72d8f00b071f\") " pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:31.763168 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:31.763145    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 22:22:31.763284 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:31.763184    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 22:22:31.763284 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:31.763198    2583 projected.go:196] Error preparing data for projected volume kube-api-access-x6lg5 for pod openshift-network-diagnostics/network-check-target-tvd42: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:31.763284 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:31.763272    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5 podName:492cf2fb-60f4-494f-8608-72d8f00b071f nodeName:}" failed. No retries permitted until 2026-07-23 22:22:35.763250492 +0000 UTC m=+10.080779281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6lg5" (UniqueName: "kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5") pod "network-check-target-tvd42" (UID: "492cf2fb-60f4-494f-8608-72d8f00b071f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:31.763774 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:31.763708    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:31.763774 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:31.763764    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret podName:729340b2-7e6e-474a-8475-db80f5a1ac8b nodeName:}" failed. No retries permitted until 2026-07-23 22:22:35.763748772 +0000 UTC m=+10.081277564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret") pod "global-pull-secret-syncer-mtsxj" (UID: "729340b2-7e6e-474a-8475-db80f5a1ac8b") : object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:31.863979 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:31.863940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:31.864209 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:31.864086    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:31.864209 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:31.864151    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs podName:6d068205-8a0a-4ae6-b8a2-f831bf0e9752 nodeName:}" failed. No retries permitted until 2026-07-23 22:22:35.864132524 +0000 UTC m=+10.181661318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs") pod "network-metrics-daemon-d6b9c" (UID: "6d068205-8a0a-4ae6-b8a2-f831bf0e9752") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:32.227786 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:32.227344    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:32.227786 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:32.227344    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:32.227786 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:32.227461    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:32.227786 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:32.227492    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:32.316041 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:32.315975    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" event={"ID":"3fa81d99861b4ec4b271cd1e93427d67","Type":"ContainerStarted","Data":"ca43e56610eb92c9e028380a72339e1729fd47722969685ab56d405a9460d256"}
Jul 23 22:22:32.334912 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:32.334342    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal" podStartSLOduration=5.334321336 podStartE2EDuration="5.334321336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:31.329326571 +0000 UTC m=+5.646855381" watchObservedRunningTime="2026-07-23 22:22:32.334321336 +0000 UTC m=+6.651850143"
Jul 23 22:22:33.223359 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:33.223325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:33.223539 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:33.223474    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:34.226790 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:34.226712    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:34.227264 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:34.226838    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:34.227264 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:34.226937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:34.227264 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:34.227049    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:34.321145 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:34.321107    2583 generic.go:356] "Generic (PLEG): container finished" podID="3fa81d99861b4ec4b271cd1e93427d67" containerID="ca43e56610eb92c9e028380a72339e1729fd47722969685ab56d405a9460d256" exitCode=0
Jul 23 22:22:34.321305 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:34.321158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" event={"ID":"3fa81d99861b4ec4b271cd1e93427d67","Type":"ContainerDied","Data":"ca43e56610eb92c9e028380a72339e1729fd47722969685ab56d405a9460d256"}
Jul 23 22:22:35.223152 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:35.223089    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:35.223329 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:35.223262    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:35.796872 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:35.796835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:35.797340 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:35.796887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lg5\" (UniqueName: \"kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5\") pod \"network-check-target-tvd42\" (UID: \"492cf2fb-60f4-494f-8608-72d8f00b071f\") " pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:35.797340 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:35.797020    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:35.797340 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:35.797107    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret podName:729340b2-7e6e-474a-8475-db80f5a1ac8b nodeName:}" failed. No retries permitted until 2026-07-23 22:22:43.797088099 +0000 UTC m=+18.114616899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret") pod "global-pull-secret-syncer-mtsxj" (UID: "729340b2-7e6e-474a-8475-db80f5a1ac8b") : object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:35.797340 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:35.797025    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 22:22:35.797340 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:35.797163    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 22:22:35.797340 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:35.797179    2583 projected.go:196] Error preparing data for projected volume kube-api-access-x6lg5 for pod openshift-network-diagnostics/network-check-target-tvd42: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:35.797340 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:35.797239    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5 podName:492cf2fb-60f4-494f-8608-72d8f00b071f nodeName:}" failed. No retries permitted until 2026-07-23 22:22:43.797225445 +0000 UTC m=+18.114754249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6lg5" (UniqueName: "kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5") pod "network-check-target-tvd42" (UID: "492cf2fb-60f4-494f-8608-72d8f00b071f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:35.897740 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:35.897690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:35.897911 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:35.897832    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:35.897997 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:35.897914    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs podName:6d068205-8a0a-4ae6-b8a2-f831bf0e9752 nodeName:}" failed. No retries permitted until 2026-07-23 22:22:43.897890223 +0000 UTC m=+18.215419011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs") pod "network-metrics-daemon-d6b9c" (UID: "6d068205-8a0a-4ae6-b8a2-f831bf0e9752") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:36.227272 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:36.226759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:36.227272 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:36.226889    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:36.227272 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:36.226949    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:36.227272 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:36.227048    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:37.223519 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:37.223480    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:37.224011 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:37.223624    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:38.225546 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:38.225491    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:38.225546 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:38.225513    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:38.226095 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:38.225673    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:38.226095 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:38.225763    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:39.222666 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:39.222641    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:39.222800 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:39.222741    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:40.222975 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:40.222915    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:40.223429 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:40.223069    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:40.223429 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:40.223125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:40.223429 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:40.223247    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:41.223118 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:41.223084    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:41.223508 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:41.223198    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:42.223386 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:42.223346    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:42.223859 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:42.223478    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:42.223859 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:42.223510    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:42.223859 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:42.223575    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:43.222863 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:43.222788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:43.223021 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:43.222912    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:43.861613 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:43.861575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lg5\" (UniqueName: \"kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5\") pod \"network-check-target-tvd42\" (UID: \"492cf2fb-60f4-494f-8608-72d8f00b071f\") " pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:43.862100 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:43.861674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:43.862100 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:43.861778    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 22:22:43.862100 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:43.861803    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 22:22:43.862100 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:43.861819    2583 projected.go:196] Error preparing data for projected volume kube-api-access-x6lg5 for pod openshift-network-diagnostics/network-check-target-tvd42: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:43.862100 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:43.861817    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:43.862100 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:43.861891    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5 podName:492cf2fb-60f4-494f-8608-72d8f00b071f nodeName:}" failed. No retries permitted until 2026-07-23 22:22:59.861872466 +0000 UTC m=+34.179401271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6lg5" (UniqueName: "kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5") pod "network-check-target-tvd42" (UID: "492cf2fb-60f4-494f-8608-72d8f00b071f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:43.862100 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:43.861915    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret podName:729340b2-7e6e-474a-8475-db80f5a1ac8b nodeName:}" failed. No retries permitted until 2026-07-23 22:22:59.86190483 +0000 UTC m=+34.179433632 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret") pod "global-pull-secret-syncer-mtsxj" (UID: "729340b2-7e6e-474a-8475-db80f5a1ac8b") : object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:43.962265 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:43.962231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:43.962424 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:43.962383    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:43.962505 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:43.962464    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs podName:6d068205-8a0a-4ae6-b8a2-f831bf0e9752 nodeName:}" failed. No retries permitted until 2026-07-23 22:22:59.962442347 +0000 UTC m=+34.279971138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs") pod "network-metrics-daemon-d6b9c" (UID: "6d068205-8a0a-4ae6-b8a2-f831bf0e9752") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:44.223844 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:44.223769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:44.224015 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:44.223891    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:44.224015 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:44.223982    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:44.224135 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:44.224080    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:45.223132 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:45.223098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:45.223540 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:45.223209    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:46.224066 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:46.224003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:46.224513 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:46.224116    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:46.224513 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:46.224175    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:46.224513 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:46.224270    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:47.223373 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:47.223339    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:47.223563 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:47.223471    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:48.222993 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.222799    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:48.223473 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.222799    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:48.223473 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:48.223076    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:48.223473 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:48.223184    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:48.349715 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.349563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" event={"ID":"3fa81d99861b4ec4b271cd1e93427d67","Type":"ContainerStarted","Data":"17208c9cf174dcf13fb227667829b97f77f9c53b02f6e6fe751beff5ef09ef33"}
Jul 23 22:22:48.351218 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.351185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2bk9g" event={"ID":"051cb605-0708-4f32-bf6e-1e687d328a0a","Type":"ContainerStarted","Data":"fc9716d942bddf5edfb943d5b275d35a751fc75006c9b069d9899f28e2deaeed"}
Jul 23 22:22:48.352617 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.352590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-68765" event={"ID":"89ffe12b-74bf-4582-8395-b47766839015","Type":"ContainerStarted","Data":"b5b16d50fdb53c7e979b9c319b872c417620b60b90e6f59c9c1b88d427b1d654"}
Jul 23 22:22:48.354210 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.354184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj" event={"ID":"012cefdf-5d65-4f60-aa38-35e098c7f8be","Type":"ContainerStarted","Data":"eeaab23d178ae9fc52dfe7242ec30a8a20bb00e750213be1d5236136ff5e7faf"}
Jul 23 22:22:48.355914 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.355891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92v68" event={"ID":"fcd3c0d0-f7ad-483b-bb05-c55094232c7a","Type":"ContainerStarted","Data":"f48794ffd7e2968f8780b0d269ab3dc76d687daf284263146f2e6c932c6e0fd2"}
Jul 23 22:22:48.356019 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.355989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92v68" event={"ID":"fcd3c0d0-f7ad-483b-bb05-c55094232c7a","Type":"ContainerStarted","Data":"344256a39f407975b0bf1102aad0d2b4797a8b0d9481c13c3a30e3f7fcc7725e"}
Jul 23 22:22:48.357282 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.357261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kwlkg" event={"ID":"68c30bcd-6fb8-4d32-adcc-881a10cdc158","Type":"ContainerStarted","Data":"b1b9636fd1fc3b1769d0990ef9d1bab54a8861fe001ce5869ea851e590e0e108"}
Jul 23 22:22:48.358623 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.358604    2583 generic.go:356] "Generic (PLEG): container finished" podID="cd941d5f-38b3-49b4-a92e-afcec5348fb2" containerID="78d9418bdf32e6b5d4662f46502354b499403bd342de1485ba4b9fe5ff7eab12" exitCode=0
Jul 23 22:22:48.358716 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.358665    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w5kzg" event={"ID":"cd941d5f-38b3-49b4-a92e-afcec5348fb2","Type":"ContainerDied","Data":"78d9418bdf32e6b5d4662f46502354b499403bd342de1485ba4b9fe5ff7eab12"}
Jul 23 22:22:48.360111 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.360090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xjxj" event={"ID":"889cb48e-740e-4093-9908-67f62a225840","Type":"ContainerStarted","Data":"c80f996ef3a77ec755f1b09ce91749a80250bdd49778365b8cec83239a093906"}
Jul 23 22:22:48.361569 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.361552    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t6h69" event={"ID":"7e4891f9-35db-4acd-9b5f-917a5d8917db","Type":"ContainerStarted","Data":"5f9965205a36e80a38395fca32372c8207ba73c773198bdd5ded5ce4b58e177e"}
Jul 23 22:22:48.366741 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.366709    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" podStartSLOduration=21.366699144 podStartE2EDuration="21.366699144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:48.366273558 +0000 UTC m=+22.683802366" watchObservedRunningTime="2026-07-23 22:22:48.366699144 +0000 UTC m=+22.684227953"
Jul 23 22:22:48.383597 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.383544    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-68765" podStartSLOduration=3.08652221 podStartE2EDuration="22.383525087s" totalImagesPullingTime="19.297002877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:48.382910511 +0000 UTC m=+22.700439321" watchObservedRunningTime="2026-07-23 22:22:48.383525087 +0000 UTC m=+22.701053896"
Jul 23 22:22:48.405029 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.404981    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t6h69" podStartSLOduration=3.061032412 podStartE2EDuration="22.404963561s" totalImagesPullingTime="19.343931149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:48.404200227 +0000 UTC m=+22.721729036" watchObservedRunningTime="2026-07-23 22:22:48.404963561 +0000 UTC m=+22.722492371"
Jul 23 22:22:48.420703 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.420653    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4xjxj" podStartSLOduration=3.081146421 podStartE2EDuration="22.420636338s" totalImagesPullingTime="19.339489917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:48.420507138 +0000 UTC m=+22.738035946" watchObservedRunningTime="2026-07-23 22:22:48.420636338 +0000 UTC m=+22.738165148"
Jul 23 22:22:48.488527 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.488473    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2bk9g" podStartSLOduration=3.186508791 podStartE2EDuration="22.488454438s" totalImagesPullingTime="19.301945647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:48.487151492 +0000 UTC m=+22.804680302" watchObservedRunningTime="2026-07-23 22:22:48.488454438 +0000 UTC m=+22.805983248"
Jul 23 22:22:48.508119 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:48.508077    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kwlkg" podStartSLOduration=3.146768433 podStartE2EDuration="22.508061434s" totalImagesPullingTime="19.361293001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:48.507762305 +0000 UTC m=+22.825291114" watchObservedRunningTime="2026-07-23 22:22:48.508061434 +0000 UTC m=+22.825590242"
Jul 23 22:22:49.205142 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:49.205114    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 22:22:49.223105 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:49.223085    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:49.223664 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:49.223182    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:49.366216 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:49.366085    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj" event={"ID":"012cefdf-5d65-4f60-aa38-35e098c7f8be","Type":"ContainerStarted","Data":"7c166a73ae591478de2e9f350bb8371fb762fbd8678e440ae60d909b56c0793d"}
Jul 23 22:22:49.367687 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:49.367660    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ss95p" event={"ID":"b82bbc6a-0156-4fa4-846f-3302c983648e","Type":"ContainerStarted","Data":"cbc6ce1d045432d15c55603f1a35878ce0f3a0b5d6c0d14bf8fec0d2a45aaff0"}
Jul 23 22:22:49.371104 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:49.371069    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92v68" event={"ID":"fcd3c0d0-f7ad-483b-bb05-c55094232c7a","Type":"ContainerStarted","Data":"903882c09b34a409c55dba33052a19ce8fbe4ccd89558c7201d6c83b2dcd4ea4"}
Jul 23 22:22:49.371205 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:49.371152    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92v68" event={"ID":"fcd3c0d0-f7ad-483b-bb05-c55094232c7a","Type":"ContainerStarted","Data":"8cdf40c74882c4412adcf1c965e7811078fa66e76c63c853ade3f8504b6c2b80"}
Jul 23 22:22:49.371205 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:49.371200    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92v68" event={"ID":"fcd3c0d0-f7ad-483b-bb05-c55094232c7a","Type":"ContainerStarted","Data":"01b1b3e9ab61d507fabe5afe705e5502b14f0c6af44f55a5d01f1b2895556b18"}
Jul 23 22:22:49.371326 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:49.371230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92v68" event={"ID":"fcd3c0d0-f7ad-483b-bb05-c55094232c7a","Type":"ContainerStarted","Data":"7b3fad5457acccad6eccbe35f922584a1edc871a65fed568538819bf82a51ddb"}
Jul 23 22:22:49.384959 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:49.384899    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ss95p" podStartSLOduration=4.056235775 podStartE2EDuration="23.3848837s" totalImagesPullingTime="19.328647925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:49.384629808 +0000 UTC m=+23.702158629" watchObservedRunningTime="2026-07-23 22:22:49.3848837 +0000 UTC m=+23.702412510"
Jul 23 22:22:50.192608 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:50.192503    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T22:22:49.20513797Z","UUID":"9811579a-4e68-4367-926a-b41a15cf1484","Handler":null,"Name":"","Endpoint":""}
Jul 23 22:22:50.194320 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:50.194285    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 22:22:50.194454 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:50.194325    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 22:22:50.227054 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:50.227012    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:50.227054 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:50.227020    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:50.227395 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:50.227111    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:50.227395 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:50.227258    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:50.376214 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:50.376172    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj" event={"ID":"012cefdf-5d65-4f60-aa38-35e098c7f8be","Type":"ContainerStarted","Data":"76866d0e53f6706d3f7941a2b3b34729eff6e02b845b0144140ce76c0e128720"}
Jul 23 22:22:50.394255 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:50.394179    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48szj" podStartSLOduration=3.095673373 podStartE2EDuration="24.394165435s" totalImagesPullingTime="21.298492062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:50.393561102 +0000 UTC m=+24.711089956" watchObservedRunningTime="2026-07-23 22:22:50.394165435 +0000 UTC m=+24.711694244"
Jul 23 22:22:51.222908 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:51.222817    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:51.223093 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:51.222965    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:51.382905 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:51.382860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92v68" event={"ID":"fcd3c0d0-f7ad-483b-bb05-c55094232c7a","Type":"ContainerStarted","Data":"98705b20c6d2c277d4cd414399c60436d089c97f8b50fe842aa447bec295d8f3"}
Jul 23 22:22:52.222802 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:52.222765    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:52.223021 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:52.222810    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:52.223021 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:52.222906    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:52.223021 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:52.222997    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:52.706089 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:52.705837    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:52.706497 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:52.706448    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:53.224801 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:53.224771    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:53.224954 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:53.224911    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:53.390583 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:53.390551    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92v68" event={"ID":"fcd3c0d0-f7ad-483b-bb05-c55094232c7a","Type":"ContainerStarted","Data":"edfc737525473b4cb6ebf6bbf7f50d3cb09160a6831f527c015e9ebdda31931b"}
Jul 23 22:22:53.390806 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:53.390785    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:53.392376 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:53.392353    2583 generic.go:356] "Generic (PLEG): container finished" podID="cd941d5f-38b3-49b4-a92e-afcec5348fb2" containerID="0132873ea498b2c2d63acfee7dcbdb18b58623270d2ffaa4ab1a469a25c1b6f6" exitCode=0
Jul 23 22:22:53.392471 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:53.392441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w5kzg" event={"ID":"cd941d5f-38b3-49b4-a92e-afcec5348fb2","Type":"ContainerDied","Data":"0132873ea498b2c2d63acfee7dcbdb18b58623270d2ffaa4ab1a469a25c1b6f6"}
Jul 23 22:22:53.406034 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:53.406016    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:53.420664 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:53.420626    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-92v68" podStartSLOduration=7.780582529 podStartE2EDuration="27.420615305s" totalImagesPullingTime="19.640032776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:53.420341559 +0000 UTC m=+27.737870368" watchObservedRunningTime="2026-07-23 22:22:53.420615305 +0000 UTC m=+27.738144114"
Jul 23 22:22:54.223684 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:54.223651    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:54.224357 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:54.223661    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:54.224357 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:54.223757    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:54.224357 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:54.223830    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:54.394985 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:54.394737    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 23 22:22:54.395143 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:54.394900    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:54.416025 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:54.415995    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:55.223206 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:55.223176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:55.223363 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:55.223287    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:55.226912 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:55.226886    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tvd42"]
Jul 23 22:22:55.227294 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:55.227002    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:55.227294 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:55.227070    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:55.230963 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:55.230935    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d6b9c"]
Jul 23 22:22:55.231405 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:55.231386    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mtsxj"]
Jul 23 22:22:55.231477 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:55.231466    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:55.231565 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:55.231545    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:55.398916 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:55.398882    2583 generic.go:356] "Generic (PLEG): container finished" podID="cd941d5f-38b3-49b4-a92e-afcec5348fb2" containerID="d46c6bece2479a9a4172b9ad9e9ab51549e5fcb500f888d33d5397b37c54fbd2" exitCode=0
Jul 23 22:22:55.399095 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:55.398965    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:55.399095 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:55.398958    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w5kzg" event={"ID":"cd941d5f-38b3-49b4-a92e-afcec5348fb2","Type":"ContainerDied","Data":"d46c6bece2479a9a4172b9ad9e9ab51549e5fcb500f888d33d5397b37c54fbd2"}
Jul 23 22:22:55.399241 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:55.399225    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 23 22:22:55.399299 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:55.399268    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:56.401063 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:56.401033    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 23 22:22:57.223323 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:57.223294    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:57.223323 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:57.223311    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:57.223532 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:57.223431    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:57.223532 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:57.223495    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:57.223532 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:57.223419    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:57.223673 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:57.223553    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:57.405458 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:57.405421    2583 generic.go:356] "Generic (PLEG): container finished" podID="cd941d5f-38b3-49b4-a92e-afcec5348fb2" containerID="3b7c1680474f21df8ab1f8012e10b2d63a9186d191898494a19f8a4b8871967c" exitCode=0
Jul 23 22:22:57.405815 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:57.405472    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w5kzg" event={"ID":"cd941d5f-38b3-49b4-a92e-afcec5348fb2","Type":"ContainerDied","Data":"3b7c1680474f21df8ab1f8012e10b2d63a9186d191898494a19f8a4b8871967c"}
Jul 23 22:22:57.794501 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:57.794351    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:22:57.912093 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:57.912058    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:57.912262 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:57.912215    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 23 22:22:57.912854 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:57.912790    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kwlkg"
Jul 23 22:22:59.222992 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:59.222958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:59.223412 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:59.222958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:59.223412 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:59.223100    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvd42" podUID="492cf2fb-60f4-494f-8608-72d8f00b071f"
Jul 23 22:22:59.223412 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:59.222958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:59.223412 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:59.223171    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6b9c" podUID="6d068205-8a0a-4ae6-b8a2-f831bf0e9752"
Jul 23 22:22:59.223412 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:59.223256    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtsxj" podUID="729340b2-7e6e-474a-8475-db80f5a1ac8b"
Jul 23 22:22:59.879515 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:59.879479    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:22:59.879515 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:59.879520    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lg5\" (UniqueName: \"kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5\") pod \"network-check-target-tvd42\" (UID: \"492cf2fb-60f4-494f-8608-72d8f00b071f\") " pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:22:59.879744 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:59.879626    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:59.879744 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:59.879682    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret podName:729340b2-7e6e-474a-8475-db80f5a1ac8b nodeName:}" failed. No retries permitted until 2026-07-23 22:23:31.879668819 +0000 UTC m=+66.197197606 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret") pod "global-pull-secret-syncer-mtsxj" (UID: "729340b2-7e6e-474a-8475-db80f5a1ac8b") : object "kube-system"/"original-pull-secret" not registered
Jul 23 22:22:59.879744 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:59.879633    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 22:22:59.879906 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:59.879746    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 22:22:59.879906 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:59.879766    2583 projected.go:196] Error preparing data for projected volume kube-api-access-x6lg5 for pod openshift-network-diagnostics/network-check-target-tvd42: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:59.879906 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:59.879821    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5 podName:492cf2fb-60f4-494f-8608-72d8f00b071f nodeName:}" failed. No retries permitted until 2026-07-23 22:23:31.879804737 +0000 UTC m=+66.197333543 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6lg5" (UniqueName: "kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5") pod "network-check-target-tvd42" (UID: "492cf2fb-60f4-494f-8608-72d8f00b071f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 22:22:59.979835 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:22:59.979803    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:22:59.980012 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:59.979991    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:22:59.980091 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:22:59.980073    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs podName:6d068205-8a0a-4ae6-b8a2-f831bf0e9752 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:31.980048544 +0000 UTC m=+66.297577349 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs") pod "network-metrics-daemon-d6b9c" (UID: "6d068205-8a0a-4ae6-b8a2-f831bf0e9752") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 22:23:00.966525 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:00.966497    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeReady"
Jul 23 22:23:00.966966 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:00.966644    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 22:23:01.006101 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.006068    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-575b6566d-rrhk9"]
Jul 23 22:23:01.010610 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.010590    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.014998 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.014974    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dk6d4\""
Jul 23 22:23:01.015115 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.015016    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 23 22:23:01.015115 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.015041    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 23 22:23:01.015629 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.015399    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 23 22:23:01.023635 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.023616    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-575b6566d-rrhk9"]
Jul 23 22:23:01.023816 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.023798    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 23 22:23:01.027459 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.027435    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-78sqp"]
Jul 23 22:23:01.030800 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.030781    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lnlzn"]
Jul 23 22:23:01.030917 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.030903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-78sqp"
Jul 23 22:23:01.034188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.034118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.037722 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.037699    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6t6v4\""
Jul 23 22:23:01.037831 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.037799    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fmr8p\""
Jul 23 22:23:01.037950 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.037827    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 22:23:01.038015 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.037949    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 22:23:01.038172 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.038106    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 22:23:01.038247 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.038227    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 22:23:01.038453 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.038424    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 22:23:01.051852 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.051829    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-78sqp"]
Jul 23 22:23:01.053364 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.053343    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lnlzn"]
Jul 23 22:23:01.187631 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187600    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.187774 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60e8cde0-a29c-4982-9833-b31d8e864630-ca-trust-extracted\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.187774 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187665    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-installation-pull-secrets\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.187774 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187681    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72439e60-4ee5-4235-bd03-68c54f40aa4e-cert\") pod \"ingress-canary-78sqp\" (UID: \"72439e60-4ee5-4235-bd03-68c54f40aa4e\") " pod="openshift-ingress-canary/ingress-canary-78sqp"
Jul 23 22:23:01.187774 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187710    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl7l9\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-kube-api-access-zl7l9\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.187774 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-tmp-dir\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.187774 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jrnp\" (UniqueName: \"kubernetes.io/projected/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-kube-api-access-6jrnp\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.187975 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187829    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znldb\" (UniqueName: \"kubernetes.io/projected/72439e60-4ee5-4235-bd03-68c54f40aa4e-kube-api-access-znldb\") pod \"ingress-canary-78sqp\" (UID: \"72439e60-4ee5-4235-bd03-68c54f40aa4e\") " pod="openshift-ingress-canary/ingress-canary-78sqp"
Jul 23 22:23:01.187975 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.187975 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187899    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-trusted-ca\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.187975 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-config-volume\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.188093 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.187990    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-bound-sa-token\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.188093 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.188011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-image-registry-private-configuration\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.188093 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.188029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-registry-certificates\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.223579 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.223527    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:23:01.223662 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.223635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:23:01.223662 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.223658    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:23:01.229624 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.229605    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 22:23:01.229795 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.229768    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 22:23:01.230064 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.230040    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-655rt\""
Jul 23 22:23:01.230161 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.230078    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 22:23:01.230161 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.230038    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 22:23:01.230161 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.230100    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2pbtn\""
Jul 23 22:23:01.288465 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znldb\" (UniqueName: \"kubernetes.io/projected/72439e60-4ee5-4235-bd03-68c54f40aa4e-kube-api-access-znldb\") pod \"ingress-canary-78sqp\" (UID: \"72439e60-4ee5-4235-bd03-68c54f40aa4e\") " pod="openshift-ingress-canary/ingress-canary-78sqp"
Jul 23 22:23:01.288576 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.288576 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-trusted-ca\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.288576 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288507    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-config-volume\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.288720 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-bound-sa-token\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.288720 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-image-registry-private-configuration\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.288720 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-registry-certificates\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.288720 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:01.288701    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 22:23:01.288720 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:01.288720    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575b6566d-rrhk9: secret "image-registry-tls" not found
Jul 23 22:23:01.288961 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.288961 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:01.288781    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls podName:60e8cde0-a29c-4982-9833-b31d8e864630 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:01.78876054 +0000 UTC m=+36.106289340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls") pod "image-registry-575b6566d-rrhk9" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630") : secret "image-registry-tls" not found
Jul 23 22:23:01.288961 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:01.288831    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 22:23:01.288961 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288839    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60e8cde0-a29c-4982-9833-b31d8e864630-ca-trust-extracted\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.288961 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-installation-pull-secrets\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.288961 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:01.288881    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls podName:3a3df64e-97c8-4a17-8848-6fd2c8fd4057 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:01.788865425 +0000 UTC m=+36.106394221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls") pod "dns-default-lnlzn" (UID: "3a3df64e-97c8-4a17-8848-6fd2c8fd4057") : secret "dns-default-metrics-tls" not found
Jul 23 22:23:01.288961 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72439e60-4ee5-4235-bd03-68c54f40aa4e-cert\") pod \"ingress-canary-78sqp\" (UID: \"72439e60-4ee5-4235-bd03-68c54f40aa4e\") " pod="openshift-ingress-canary/ingress-canary-78sqp"
Jul 23 22:23:01.289306 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl7l9\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-kube-api-access-zl7l9\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.289306 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.288995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-tmp-dir\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.289306 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.289020    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jrnp\" (UniqueName: \"kubernetes.io/projected/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-kube-api-access-6jrnp\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.289306 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.289234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-registry-certificates\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.289306 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.289291    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-config-volume\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.289547 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.289424    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-tmp-dir\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.289547 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.289486    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60e8cde0-a29c-4982-9833-b31d8e864630-ca-trust-extracted\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.289633 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.289621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-trusted-ca\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.292587 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.292555    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72439e60-4ee5-4235-bd03-68c54f40aa4e-cert\") pod \"ingress-canary-78sqp\" (UID: \"72439e60-4ee5-4235-bd03-68c54f40aa4e\") " pod="openshift-ingress-canary/ingress-canary-78sqp"
Jul 23 22:23:01.292587 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.292570    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-image-registry-private-configuration\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.292702 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.292577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-installation-pull-secrets\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.297850 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.297823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znldb\" (UniqueName: \"kubernetes.io/projected/72439e60-4ee5-4235-bd03-68c54f40aa4e-kube-api-access-znldb\") pod \"ingress-canary-78sqp\" (UID: \"72439e60-4ee5-4235-bd03-68c54f40aa4e\") " pod="openshift-ingress-canary/ingress-canary-78sqp"
Jul 23 22:23:01.298794 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.298137    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-bound-sa-token\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.298794 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.298395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl7l9\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-kube-api-access-zl7l9\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.301871 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.301846    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jrnp\" (UniqueName: \"kubernetes.io/projected/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-kube-api-access-6jrnp\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.341952 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.341902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-78sqp"
Jul 23 22:23:01.420684 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.420356    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w5kzg" event={"ID":"cd941d5f-38b3-49b4-a92e-afcec5348fb2","Type":"ContainerStarted","Data":"6e759b5091ebcc252d56ac2d9b1b9d11e5511181939dfae38b0deef09d008fed"}
Jul 23 22:23:01.477829 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.477725    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-78sqp"]
Jul 23 22:23:01.481420 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:01.481396    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72439e60_4ee5_4235_bd03_68c54f40aa4e.slice/crio-6df777d6a9956671f2d5ed853c1b11623ba6cb261dab934e2391764659b2e79f WatchSource:0}: Error finding container 6df777d6a9956671f2d5ed853c1b11623ba6cb261dab934e2391764659b2e79f: Status 404 returned error can't find the container with id 6df777d6a9956671f2d5ed853c1b11623ba6cb261dab934e2391764659b2e79f
Jul 23 22:23:01.792327 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.792233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:01.792327 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:01.792318    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:01.792543 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:01.792396    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 22:23:01.792543 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:01.792415    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575b6566d-rrhk9: secret "image-registry-tls" not found
Jul 23 22:23:01.792543 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:01.792444    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 22:23:01.792543 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:01.792473    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls podName:60e8cde0-a29c-4982-9833-b31d8e864630 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:02.792456597 +0000 UTC m=+37.109985384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls") pod "image-registry-575b6566d-rrhk9" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630") : secret "image-registry-tls" not found
Jul 23 22:23:01.792543 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:01.792494    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls podName:3a3df64e-97c8-4a17-8848-6fd2c8fd4057 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:02.79247946 +0000 UTC m=+37.110008254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls") pod "dns-default-lnlzn" (UID: "3a3df64e-97c8-4a17-8848-6fd2c8fd4057") : secret "dns-default-metrics-tls" not found
Jul 23 22:23:02.425205 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:02.425171    2583 generic.go:356] "Generic (PLEG): container finished" podID="cd941d5f-38b3-49b4-a92e-afcec5348fb2" containerID="6e759b5091ebcc252d56ac2d9b1b9d11e5511181939dfae38b0deef09d008fed" exitCode=0
Jul 23 22:23:02.426022 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:02.425254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w5kzg" event={"ID":"cd941d5f-38b3-49b4-a92e-afcec5348fb2","Type":"ContainerDied","Data":"6e759b5091ebcc252d56ac2d9b1b9d11e5511181939dfae38b0deef09d008fed"}
Jul 23 22:23:02.426442 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:02.426394    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-78sqp" event={"ID":"72439e60-4ee5-4235-bd03-68c54f40aa4e","Type":"ContainerStarted","Data":"6df777d6a9956671f2d5ed853c1b11623ba6cb261dab934e2391764659b2e79f"}
Jul 23 22:23:02.799762 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:02.799674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:02.799956 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:02.799748    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:02.799956 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:02.799838    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 22:23:02.799956 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:02.799898    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls podName:3a3df64e-97c8-4a17-8848-6fd2c8fd4057 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:04.799885546 +0000 UTC m=+39.117414337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls") pod "dns-default-lnlzn" (UID: "3a3df64e-97c8-4a17-8848-6fd2c8fd4057") : secret "dns-default-metrics-tls" not found
Jul 23 22:23:02.799956 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:02.799841    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 22:23:02.799956 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:02.799958    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575b6566d-rrhk9: secret "image-registry-tls" not found
Jul 23 22:23:02.800176 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:02.800000    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls podName:60e8cde0-a29c-4982-9833-b31d8e864630 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:04.799988748 +0000 UTC m=+39.117517536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls") pod "image-registry-575b6566d-rrhk9" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630") : secret "image-registry-tls" not found
Jul 23 22:23:03.432260 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:03.432229    2583 generic.go:356] "Generic (PLEG): container finished" podID="cd941d5f-38b3-49b4-a92e-afcec5348fb2" containerID="cc8122fe738698cab604b0828dc4ef669fab43df5cad13bd05fe50b067d9529e" exitCode=0
Jul 23 22:23:03.432672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:03.432269    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w5kzg" event={"ID":"cd941d5f-38b3-49b4-a92e-afcec5348fb2","Type":"ContainerDied","Data":"cc8122fe738698cab604b0828dc4ef669fab43df5cad13bd05fe50b067d9529e"}
Jul 23 22:23:04.438858 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:04.438603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-78sqp" event={"ID":"72439e60-4ee5-4235-bd03-68c54f40aa4e","Type":"ContainerStarted","Data":"a80cf257fbdf296d3b030c8270dee7b3ea52de6f7125c0fa111ce365cb15bff5"}
Jul 23 22:23:04.442020 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:04.441998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w5kzg" event={"ID":"cd941d5f-38b3-49b4-a92e-afcec5348fb2","Type":"ContainerStarted","Data":"97512215e27d863f9fec4802c9d7650453c08b98977b3fafcd05b17cae297228"}
Jul 23 22:23:04.457565 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:04.457522    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-78sqp" podStartSLOduration=0.857110336 podStartE2EDuration="3.457506978s" totalImagesPullingTime="2.600396642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:04.457465761 +0000 UTC m=+38.774994572" watchObservedRunningTime="2026-07-23 22:23:04.457506978 +0000 UTC m=+38.775035790"
Jul 23 22:23:04.482694 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:04.482646    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w5kzg" podStartSLOduration=8.756477936 podStartE2EDuration="38.48263212s" totalImagesPullingTime="29.726154184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:04.482050538 +0000 UTC m=+38.799579354" watchObservedRunningTime="2026-07-23 22:23:04.48263212 +0000 UTC m=+38.800160929"
Jul 23 22:23:04.634843 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:04.634805    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwlkg_68c30bcd-6fb8-4d32-adcc-881a10cdc158/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:04.730174 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:04.730146    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwlkg_68c30bcd-6fb8-4d32-adcc-881a10cdc158/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:04.815250 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:04.815214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:04.815402 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:04.815295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:04.815402 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:04.815375    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 22:23:04.815402 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:04.815392    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575b6566d-rrhk9: secret "image-registry-tls" not found
Jul 23 22:23:04.815515 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:04.815412    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 22:23:04.815515 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:04.815471    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls podName:60e8cde0-a29c-4982-9833-b31d8e864630 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:08.815452606 +0000 UTC m=+43.132981395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls") pod "image-registry-575b6566d-rrhk9" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630") : secret "image-registry-tls" not found
Jul 23 22:23:04.815515 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:04.815490    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls podName:3a3df64e-97c8-4a17-8848-6fd2c8fd4057 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:08.815481759 +0000 UTC m=+43.133010546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls") pod "dns-default-lnlzn" (UID: "3a3df64e-97c8-4a17-8848-6fd2c8fd4057") : secret "dns-default-metrics-tls" not found
Jul 23 22:23:06.152356 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:06.152322    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwlkg_68c30bcd-6fb8-4d32-adcc-881a10cdc158/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:07.868697 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.868661    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"]
Jul 23 22:23:07.909438 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.909408    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-85wbm"]
Jul 23 22:23:07.909575 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.909525    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:07.912322 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.912299    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 23 22:23:07.912450 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.912417    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 23 22:23:07.912450 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.912429    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 23 22:23:07.912563 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.912523    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-498l9\""
Jul 23 22:23:07.933326 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.933307    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"]
Jul 23 22:23:07.933423 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.933412    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-85wbm"
Jul 23 22:23:07.935936 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.935901    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 23 22:23:07.935936 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.935917    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 23 22:23:07.936103 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.935969    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-c26qk\""
Jul 23 22:23:07.957332 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.957312    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-gbh2j"]
Jul 23 22:23:07.957438 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.957424    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:07.960070 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.960052    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 22:23:07.960177 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.960126    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 22:23:07.961177 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.961156    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 23 22:23:07.961264 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.961184    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 23 22:23:07.961264 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.961259    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-44gg6\""
Jul 23 22:23:07.977420 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.977401    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-tnv5g"]
Jul 23 22:23:07.977543 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.977528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:07.980015 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.979998    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 23 22:23:07.980097 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.980021    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wmjbs\""
Jul 23 22:23:07.980998 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.980978    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 22:23:07.981219 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.981190    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 22:23:07.981219 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.981200    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 23 22:23:07.987092 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.987075    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 23 22:23:07.997549 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.997528    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"]
Jul 23 22:23:07.997638 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.997559    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-85wbm"]
Jul 23 22:23:07.997638 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.997574    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"]
Jul 23 22:23:07.997638 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.997593    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-gbh2j"]
Jul 23 22:23:07.997638 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.997611    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-tnv5g"]
Jul 23 22:23:07.997638 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.997634    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-vv74w"]
Jul 23 22:23:07.997881 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:07.997645    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.000151 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.000131    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 23 22:23:08.000285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.000267    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 23 22:23:08.000379 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.000361    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 23 22:23:08.000452 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.000134    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 23 22:23:08.000506 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.000458    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-52sf4\""
Jul 23 22:23:08.006592 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.006564    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 23 22:23:08.011618 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.011599    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"]
Jul 23 22:23:08.011741 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.011727    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vv74w"
Jul 23 22:23:08.014160 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.014143    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wmbgv\""
Jul 23 22:23:08.033866 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.033843    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"]
Jul 23 22:23:08.033998 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.033944    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"
Jul 23 22:23:08.036697 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.036676    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 23 22:23:08.036819 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.036719    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xp764\""
Jul 23 22:23:08.036819 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.036760    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 23 22:23:08.036819 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.036773    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 23 22:23:08.036819 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.036719    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 23 22:23:08.037429 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.037410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsnv4\" (UniqueName: \"kubernetes.io/projected/5ae231e2-f5bf-4933-ad63-9c81b4931ce2-kube-api-access-jsnv4\") pod \"volume-data-source-validator-78cf4c6f6-85wbm\" (UID: \"5ae231e2-f5bf-4933-ad63-9c81b4931ce2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-85wbm"
Jul 23 22:23:08.037506 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.037479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rrqfl\" (UID: \"3bb5c7c0-a03e-48b6-b948-993b360abcb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:08.037558 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.037540    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shst7\" (UniqueName: \"kubernetes.io/projected/3bb5c7c0-a03e-48b6-b948-993b360abcb3-kube-api-access-shst7\") pod \"cluster-samples-operator-79b7c55f87-rrqfl\" (UID: \"3bb5c7c0-a03e-48b6-b948-993b360abcb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:08.053791 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.053768    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-84db8994fd-csdvv"]
Jul 23 22:23:08.053908 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.053892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"
Jul 23 22:23:08.056597 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.056577    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 23 22:23:08.056597 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.056589    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 23 22:23:08.056749 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.056624    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 23 22:23:08.056876 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.056862    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 23 22:23:08.056955 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.056899    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hrqzj\""
Jul 23 22:23:08.063882 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.063863    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-vv74w"]
Jul 23 22:23:08.063979 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.063898    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"]
Jul 23 22:23:08.063979 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.063935    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"]
Jul 23 22:23:08.063979 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.063956    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84db8994fd-csdvv"]
Jul 23 22:23:08.064080 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.064019    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.066674 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.066657    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 23 22:23:08.066900 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.066878    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-h52z8\""
Jul 23 22:23:08.066998 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.066901    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 22:23:08.066998 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.066910    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 23 22:23:08.067098 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.066901    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 23 22:23:08.067098 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.066955    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 22:23:08.067203 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.067187    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 23 22:23:08.073336 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.073316    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"]
Jul 23 22:23:08.090134 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.090114    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"]
Jul 23 22:23:08.090217 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.090197    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:08.092895 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.092874    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 23 22:23:08.093014 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.092999    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4lp4n\""
Jul 23 22:23:08.093125 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.093110    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 23 22:23:08.138639 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rrqfl\" (UID: \"3bb5c7c0-a03e-48b6-b948-993b360abcb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:08.138639 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138616    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/031164ea-f2ff-4883-80bb-30979f504bb6-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:08.138769 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138645    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c5kz\" (UniqueName: \"kubernetes.io/projected/4b018493-e500-4c1e-9b5c-88285f6a9dc4-kube-api-access-7c5kz\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.138769 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138669    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b29adb35-eeca-46dd-b18e-8a3cb48d4769-serving-cert\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.138769 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfddt\" (UniqueName: \"kubernetes.io/projected/031164ea-f2ff-4883-80bb-30979f504bb6-kube-api-access-tfddt\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:08.138769 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138710    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b018493-e500-4c1e-9b5c-88285f6a9dc4-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.138769 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.138720    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 23 22:23:08.138946 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.138797    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls podName:3bb5c7c0-a03e-48b6-b948-993b360abcb3 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:08.638769571 +0000 UTC m=+42.956298358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rrqfl" (UID: "3bb5c7c0-a03e-48b6-b948-993b360abcb3") : secret "samples-operator-tls" not found
Jul 23 22:23:08.138946 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e2ff89d-860c-44a9-b40b-983d106daea3-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6pdgh\" (UID: \"1e2ff89d-860c-44a9-b40b-983d106daea3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"
Jul 23 22:23:08.138946 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b018493-e500-4c1e-9b5c-88285f6a9dc4-serving-cert\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.138946 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnf85\" (UniqueName: \"kubernetes.io/projected/1e2ff89d-860c-44a9-b40b-983d106daea3-kube-api-access-lnf85\") pod \"kube-storage-version-migrator-operator-7658d9b749-6pdgh\" (UID: \"1e2ff89d-860c-44a9-b40b-983d106daea3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"
Jul 23 22:23:08.138946 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e2ff89d-860c-44a9-b40b-983d106daea3-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6pdgh\" (UID: \"1e2ff89d-860c-44a9-b40b-983d106daea3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"
Jul 23 22:23:08.138946 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b018493-e500-4c1e-9b5c-88285f6a9dc4-tmp\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.138946 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8897s\" (UniqueName: \"kubernetes.io/projected/b29adb35-eeca-46dd-b18e-8a3cb48d4769-kube-api-access-8897s\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.138946 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:08.139285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bft2f\" (UniqueName: \"kubernetes.io/projected/bfba70cc-edef-4788-acd6-258f2a4a15c6-kube-api-access-bft2f\") pod \"network-check-source-7bc5b6bf85-vv74w\" (UID: \"bfba70cc-edef-4788-acd6-258f2a4a15c6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vv74w"
Jul 23 22:23:08.139285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.138984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shst7\" (UniqueName: \"kubernetes.io/projected/3bb5c7c0-a03e-48b6-b948-993b360abcb3-kube-api-access-shst7\") pod \"cluster-samples-operator-79b7c55f87-rrqfl\" (UID: \"3bb5c7c0-a03e-48b6-b948-993b360abcb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:08.139285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.139008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b29adb35-eeca-46dd-b18e-8a3cb48d4769-trusted-ca\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.139285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.139090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsnv4\" (UniqueName: \"kubernetes.io/projected/5ae231e2-f5bf-4933-ad63-9c81b4931ce2-kube-api-access-jsnv4\") pod \"volume-data-source-validator-78cf4c6f6-85wbm\" (UID: \"5ae231e2-f5bf-4933-ad63-9c81b4931ce2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-85wbm"
Jul 23 22:23:08.139285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.139113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b29adb35-eeca-46dd-b18e-8a3cb48d4769-config\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.139285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.139208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4b018493-e500-4c1e-9b5c-88285f6a9dc4-snapshots\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.139285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.139230    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b018493-e500-4c1e-9b5c-88285f6a9dc4-service-ca-bundle\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.150074 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.150054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shst7\" (UniqueName: \"kubernetes.io/projected/3bb5c7c0-a03e-48b6-b948-993b360abcb3-kube-api-access-shst7\") pod \"cluster-samples-operator-79b7c55f87-rrqfl\" (UID: \"3bb5c7c0-a03e-48b6-b948-993b360abcb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:08.150074 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.150069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsnv4\" (UniqueName: \"kubernetes.io/projected/5ae231e2-f5bf-4933-ad63-9c81b4931ce2-kube-api-access-jsnv4\") pod \"volume-data-source-validator-78cf4c6f6-85wbm\" (UID: \"5ae231e2-f5bf-4933-ad63-9c81b4931ce2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-85wbm"
Jul 23 22:23:08.207796 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.207768    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwlkg_68c30bcd-6fb8-4d32-adcc-881a10cdc158/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:08.240363 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szkrs\" (UniqueName: \"kubernetes.io/projected/400dc06d-48bd-4183-afca-09e259ca6d48-kube-api-access-szkrs\") pod \"service-ca-operator-5bcbc94c78-dtpzf\" (UID: \"400dc06d-48bd-4183-afca-09e259ca6d48\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"
Jul 23 22:23:08.240472 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240368    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/031164ea-f2ff-4883-80bb-30979f504bb6-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:08.240472 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240395    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c5kz\" (UniqueName: \"kubernetes.io/projected/4b018493-e500-4c1e-9b5c-88285f6a9dc4-kube-api-access-7c5kz\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.240472 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240439    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:08.240566 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240478    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/400dc06d-48bd-4183-afca-09e259ca6d48-config\") pod \"service-ca-operator-5bcbc94c78-dtpzf\" (UID: \"400dc06d-48bd-4183-afca-09e259ca6d48\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"
Jul 23 22:23:08.240566 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b29adb35-eeca-46dd-b18e-8a3cb48d4769-serving-cert\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.240566 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfddt\" (UniqueName: \"kubernetes.io/projected/031164ea-f2ff-4883-80bb-30979f504bb6-kube-api-access-tfddt\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:08.240566 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b018493-e500-4c1e-9b5c-88285f6a9dc4-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.240738 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e2ff89d-860c-44a9-b40b-983d106daea3-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6pdgh\" (UID: \"1e2ff89d-860c-44a9-b40b-983d106daea3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"
Jul 23 22:23:08.240738 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240591    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:08.240738 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b018493-e500-4c1e-9b5c-88285f6a9dc4-serving-cert\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.240738 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnf85\" (UniqueName: \"kubernetes.io/projected/1e2ff89d-860c-44a9-b40b-983d106daea3-kube-api-access-lnf85\") pod \"kube-storage-version-migrator-operator-7658d9b749-6pdgh\" (UID: \"1e2ff89d-860c-44a9-b40b-983d106daea3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"
Jul 23 22:23:08.240738 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e2ff89d-860c-44a9-b40b-983d106daea3-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6pdgh\" (UID: \"1e2ff89d-860c-44a9-b40b-983d106daea3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"
Jul 23 22:23:08.240738 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-default-certificate\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.240738 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw7hj\" (UniqueName: \"kubernetes.io/projected/f88b4c99-967b-4ded-8e2d-00bf67380e09-kube-api-access-bw7hj\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.241089 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b018493-e500-4c1e-9b5c-88285f6a9dc4-tmp\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.241089 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.241089 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8897s\" (UniqueName: \"kubernetes.io/projected/b29adb35-eeca-46dd-b18e-8a3cb48d4769-kube-api-access-8897s\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.241089 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:08.241089 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-stats-auth\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.241089 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.240970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bft2f\" (UniqueName: \"kubernetes.io/projected/bfba70cc-edef-4788-acd6-258f2a4a15c6-kube-api-access-bft2f\") pod \"network-check-source-7bc5b6bf85-vv74w\" (UID: \"bfba70cc-edef-4788-acd6-258f2a4a15c6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vv74w"
Jul 23 22:23:08.241424 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.241140    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:08.241424 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.241207    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls podName:031164ea-f2ff-4883-80bb-30979f504bb6 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:08.741189159 +0000 UTC m=+43.058717960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-h2t79" (UID: "031164ea-f2ff-4883-80bb-30979f504bb6") : secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:08.241424 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/031164ea-f2ff-4883-80bb-30979f504bb6-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:08.241424 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b29adb35-eeca-46dd-b18e-8a3cb48d4769-trusted-ca\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.241424 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-85wbm"
Jul 23 22:23:08.241424 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.241424 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b29adb35-eeca-46dd-b18e-8a3cb48d4769-config\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.241749 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e2ff89d-860c-44a9-b40b-983d106daea3-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-6pdgh\" (UID: \"1e2ff89d-860c-44a9-b40b-983d106daea3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"
Jul 23 22:23:08.241749 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4b018493-e500-4c1e-9b5c-88285f6a9dc4-snapshots\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.241749 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b018493-e500-4c1e-9b5c-88285f6a9dc4-service-ca-bundle\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.241749 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241516    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b018493-e500-4c1e-9b5c-88285f6a9dc4-tmp\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.241749 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241599    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b018493-e500-4c1e-9b5c-88285f6a9dc4-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.241749 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/400dc06d-48bd-4183-afca-09e259ca6d48-serving-cert\") pod \"service-ca-operator-5bcbc94c78-dtpzf\" (UID: \"400dc06d-48bd-4183-afca-09e259ca6d48\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"
Jul 23 22:23:08.242062 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.241971    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4b018493-e500-4c1e-9b5c-88285f6a9dc4-snapshots\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.242062 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.242038    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b018493-e500-4c1e-9b5c-88285f6a9dc4-service-ca-bundle\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.242372 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.242349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b29adb35-eeca-46dd-b18e-8a3cb48d4769-trusted-ca\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.242372 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.242358    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b29adb35-eeca-46dd-b18e-8a3cb48d4769-config\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.243086 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.243069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e2ff89d-860c-44a9-b40b-983d106daea3-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-6pdgh\" (UID: \"1e2ff89d-860c-44a9-b40b-983d106daea3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"
Jul 23 22:23:08.243232 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.243213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b018493-e500-4c1e-9b5c-88285f6a9dc4-serving-cert\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.243866 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.243851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b29adb35-eeca-46dd-b18e-8a3cb48d4769-serving-cert\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.251611 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.251590    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bft2f\" (UniqueName: \"kubernetes.io/projected/bfba70cc-edef-4788-acd6-258f2a4a15c6-kube-api-access-bft2f\") pod \"network-check-source-7bc5b6bf85-vv74w\" (UID: \"bfba70cc-edef-4788-acd6-258f2a4a15c6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vv74w"
Jul 23 22:23:08.252737 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.252711    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c5kz\" (UniqueName: \"kubernetes.io/projected/4b018493-e500-4c1e-9b5c-88285f6a9dc4-kube-api-access-7c5kz\") pod \"insights-operator-555d9d7885-gbh2j\" (UID: \"4b018493-e500-4c1e-9b5c-88285f6a9dc4\") " pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.252912 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.252892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnf85\" (UniqueName: \"kubernetes.io/projected/1e2ff89d-860c-44a9-b40b-983d106daea3-kube-api-access-lnf85\") pod \"kube-storage-version-migrator-operator-7658d9b749-6pdgh\" (UID: \"1e2ff89d-860c-44a9-b40b-983d106daea3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"
Jul 23 22:23:08.252996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.252976    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8897s\" (UniqueName: \"kubernetes.io/projected/b29adb35-eeca-46dd-b18e-8a3cb48d4769-kube-api-access-8897s\") pod \"console-operator-565b7b569d-tnv5g\" (UID: \"b29adb35-eeca-46dd-b18e-8a3cb48d4769\") " pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.253365 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.253348    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfddt\" (UniqueName: \"kubernetes.io/projected/031164ea-f2ff-4883-80bb-30979f504bb6-kube-api-access-tfddt\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:08.287304 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.287283    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-gbh2j"
Jul 23 22:23:08.302110 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.302061    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwlkg_68c30bcd-6fb8-4d32-adcc-881a10cdc158/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:08.307887 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.307858    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:08.320496 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.320471    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vv74w"
Jul 23 22:23:08.342167 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.341977    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"
Jul 23 22:23:08.342556 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.342530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szkrs\" (UniqueName: \"kubernetes.io/projected/400dc06d-48bd-4183-afca-09e259ca6d48-kube-api-access-szkrs\") pod \"service-ca-operator-5bcbc94c78-dtpzf\" (UID: \"400dc06d-48bd-4183-afca-09e259ca6d48\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"
Jul 23 22:23:08.342642 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.342591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:08.342642 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.342634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/400dc06d-48bd-4183-afca-09e259ca6d48-config\") pod \"service-ca-operator-5bcbc94c78-dtpzf\" (UID: \"400dc06d-48bd-4183-afca-09e259ca6d48\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"
Jul 23 22:23:08.342702 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.342672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:08.342736 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.342709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-default-certificate\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.342771 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.342737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw7hj\" (UniqueName: \"kubernetes.io/projected/f88b4c99-967b-4ded-8e2d-00bf67380e09-kube-api-access-bw7hj\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.342804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.342769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.342847 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.342812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-stats-auth\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.342879 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.342855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.344270 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.342980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/400dc06d-48bd-4183-afca-09e259ca6d48-serving-cert\") pod \"service-ca-operator-5bcbc94c78-dtpzf\" (UID: \"400dc06d-48bd-4183-afca-09e259ca6d48\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"
Jul 23 22:23:08.344270 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.343014    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 22:23:08.344270 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.343081    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert podName:36d92e8d-d0d3-4b4a-aa52-f67a77ccea41 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:08.843066365 +0000 UTC m=+43.160595155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tfzqx" (UID: "36d92e8d-d0d3-4b4a-aa52-f67a77ccea41") : secret "networking-console-plugin-cert" not found
Jul 23 22:23:08.344270 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.343846    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 22:23:08.344270 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.343901    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:08.843883844 +0000 UTC m=+43.161412636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : secret "router-metrics-certs-default" not found
Jul 23 22:23:08.344270 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.343993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/400dc06d-48bd-4183-afca-09e259ca6d48-config\") pod \"service-ca-operator-5bcbc94c78-dtpzf\" (UID: \"400dc06d-48bd-4183-afca-09e259ca6d48\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"
Jul 23 22:23:08.344270 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.344138    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:08.344684 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.344325    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:08.844308966 +0000 UTC m=+43.161837766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : configmap references non-existent config key: service-ca.crt
Jul 23 22:23:08.346187 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.346137    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/400dc06d-48bd-4183-afca-09e259ca6d48-serving-cert\") pod \"service-ca-operator-5bcbc94c78-dtpzf\" (UID: \"400dc06d-48bd-4183-afca-09e259ca6d48\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"
Jul 23 22:23:08.347801 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.347298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-stats-auth\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.347801 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.347690    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-default-certificate\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.356888 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.356859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szkrs\" (UniqueName: \"kubernetes.io/projected/400dc06d-48bd-4183-afca-09e259ca6d48-kube-api-access-szkrs\") pod \"service-ca-operator-5bcbc94c78-dtpzf\" (UID: \"400dc06d-48bd-4183-afca-09e259ca6d48\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"
Jul 23 22:23:08.357764 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.357730    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw7hj\" (UniqueName: \"kubernetes.io/projected/f88b4c99-967b-4ded-8e2d-00bf67380e09-kube-api-access-bw7hj\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.363386 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.363326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"
Jul 23 22:23:08.402745 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.402708    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwlkg_68c30bcd-6fb8-4d32-adcc-881a10cdc158/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:08.418996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.418903    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-85wbm"]
Jul 23 22:23:08.423194 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:08.423119    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae231e2_f5bf_4933_ad63_9c81b4931ce2.slice/crio-2663ce9ab5276a2911a65773bdc5f6bfc8aa77cbf55f7949e8b7661a4fbc7392 WatchSource:0}: Error finding container 2663ce9ab5276a2911a65773bdc5f6bfc8aa77cbf55f7949e8b7661a4fbc7392: Status 404 returned error can't find the container with id 2663ce9ab5276a2911a65773bdc5f6bfc8aa77cbf55f7949e8b7661a4fbc7392
Jul 23 22:23:08.465361 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.463795    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-gbh2j"]
Jul 23 22:23:08.479990 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.479636    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-85wbm" event={"ID":"5ae231e2-f5bf-4933-ad63-9c81b4931ce2","Type":"ContainerStarted","Data":"2663ce9ab5276a2911a65773bdc5f6bfc8aa77cbf55f7949e8b7661a4fbc7392"}
Jul 23 22:23:08.484971 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.484369    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwlkg_68c30bcd-6fb8-4d32-adcc-881a10cdc158/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:08.551162 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.551132    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-tnv5g"]
Jul 23 22:23:08.555571 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:08.555538    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29adb35_eeca_46dd_b18e_8a3cb48d4769.slice/crio-d596bfa9c523d227df5ff76c715ee8c2981654829fd58b08277a27ab42b0108a WatchSource:0}: Error finding container d596bfa9c523d227df5ff76c715ee8c2981654829fd58b08277a27ab42b0108a: Status 404 returned error can't find the container with id d596bfa9c523d227df5ff76c715ee8c2981654829fd58b08277a27ab42b0108a
Jul 23 22:23:08.582381 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.582351    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-vv74w"]
Jul 23 22:23:08.596756 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.596732    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh"]
Jul 23 22:23:08.599432 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:08.599403    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2ff89d_860c_44a9_b40b_983d106daea3.slice/crio-efb911b885efe9825c2d05fd691a5fb571797593663bfdf6c584dcdbf05635b7 WatchSource:0}: Error finding container efb911b885efe9825c2d05fd691a5fb571797593663bfdf6c584dcdbf05635b7: Status 404 returned error can't find the container with id efb911b885efe9825c2d05fd691a5fb571797593663bfdf6c584dcdbf05635b7
Jul 23 22:23:08.618734 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.618710    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf"]
Jul 23 22:23:08.620986 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:08.620964    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400dc06d_48bd_4183_afca_09e259ca6d48.slice/crio-1968f96632ebf716546bb1739e626087575ca64903f32f1b5ebf0836361994cf WatchSource:0}: Error finding container 1968f96632ebf716546bb1739e626087575ca64903f32f1b5ebf0836361994cf: Status 404 returned error can't find the container with id 1968f96632ebf716546bb1739e626087575ca64903f32f1b5ebf0836361994cf
Jul 23 22:23:08.645787 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.645732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rrqfl\" (UID: \"3bb5c7c0-a03e-48b6-b948-993b360abcb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:08.645864 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.645841    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 23 22:23:08.645902 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.645886    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls podName:3bb5c7c0-a03e-48b6-b948-993b360abcb3 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:09.645872592 +0000 UTC m=+43.963401379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rrqfl" (UID: "3bb5c7c0-a03e-48b6-b948-993b360abcb3") : secret "samples-operator-tls" not found
Jul 23 22:23:08.746534 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.746504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:08.746661 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.746642    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:08.746714 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.746704    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls podName:031164ea-f2ff-4883-80bb-30979f504bb6 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:09.746689975 +0000 UTC m=+44.064218762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-h2t79" (UID: "031164ea-f2ff-4883-80bb-30979f504bb6") : secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:08.846992 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.846953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:08.847140 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.847012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:08.847140 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.847051    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.847140 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.847100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:08.847140 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.847105    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 22:23:08.847140 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.847126    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575b6566d-rrhk9: secret "image-registry-tls" not found
Jul 23 22:23:08.847348 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.847158    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 22:23:08.847348 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.847175    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls podName:60e8cde0-a29c-4982-9833-b31d8e864630 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:16.847160188 +0000 UTC m=+51.164688975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls") pod "image-registry-575b6566d-rrhk9" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630") : secret "image-registry-tls" not found
Jul 23 22:23:08.847348 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.847107    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 22:23:08.847348 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:08.847160    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:08.847348 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.847220    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 22:23:08.847348 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.847211    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:09.847195442 +0000 UTC m=+44.164724256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : secret "router-metrics-certs-default" not found
Jul 23 22:23:08.847348 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.847283    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls podName:3a3df64e-97c8-4a17-8848-6fd2c8fd4057 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:16.847262166 +0000 UTC m=+51.164790968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls") pod "dns-default-lnlzn" (UID: "3a3df64e-97c8-4a17-8848-6fd2c8fd4057") : secret "dns-default-metrics-tls" not found
Jul 23 22:23:08.847348 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.847315    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert podName:36d92e8d-d0d3-4b4a-aa52-f67a77ccea41 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:09.847295717 +0000 UTC m=+44.164824510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tfzqx" (UID: "36d92e8d-d0d3-4b4a-aa52-f67a77ccea41") : secret "networking-console-plugin-cert" not found
Jul 23 22:23:08.847348 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:08.847338    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:09.847329231 +0000 UTC m=+44.164858020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : configmap references non-existent config key: service-ca.crt
Jul 23 22:23:09.487200 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:09.487125    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh" event={"ID":"1e2ff89d-860c-44a9-b40b-983d106daea3","Type":"ContainerStarted","Data":"efb911b885efe9825c2d05fd691a5fb571797593663bfdf6c584dcdbf05635b7"}
Jul 23 22:23:09.489369 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:09.489337    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vv74w" event={"ID":"bfba70cc-edef-4788-acd6-258f2a4a15c6","Type":"ContainerStarted","Data":"25fcca90af26c4531764af3ee8d38e4aa31f6fa0d7d5ca4bd682ea10cac2d2f3"}
Jul 23 22:23:09.490912 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:09.490886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf" event={"ID":"400dc06d-48bd-4183-afca-09e259ca6d48","Type":"ContainerStarted","Data":"1968f96632ebf716546bb1739e626087575ca64903f32f1b5ebf0836361994cf"}
Jul 23 22:23:09.494059 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:09.494032    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g" event={"ID":"b29adb35-eeca-46dd-b18e-8a3cb48d4769","Type":"ContainerStarted","Data":"d596bfa9c523d227df5ff76c715ee8c2981654829fd58b08277a27ab42b0108a"}
Jul 23 22:23:09.496578 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:09.496554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-gbh2j" event={"ID":"4b018493-e500-4c1e-9b5c-88285f6a9dc4","Type":"ContainerStarted","Data":"2a2d95fb7fd93aa4bae74c892aea5cd6cb8e283bcab3dc0317cd0d2571b5841e"}
Jul 23 22:23:09.654346 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:09.654303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rrqfl\" (UID: \"3bb5c7c0-a03e-48b6-b948-993b360abcb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:09.654508 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:09.654443    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 23 22:23:09.654563 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:09.654521    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls podName:3bb5c7c0-a03e-48b6-b948-993b360abcb3 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:11.654500445 +0000 UTC m=+45.972029233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rrqfl" (UID: "3bb5c7c0-a03e-48b6-b948-993b360abcb3") : secret "samples-operator-tls" not found
Jul 23 22:23:09.756238 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:09.755428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:09.756238 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:09.755720    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:09.756238 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:09.755783    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls podName:031164ea-f2ff-4883-80bb-30979f504bb6 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:11.755764665 +0000 UTC m=+46.073293468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-h2t79" (UID: "031164ea-f2ff-4883-80bb-30979f504bb6") : secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:09.857861 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:09.856955    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:09.857861 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:09.857018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:09.857861 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:09.857067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:09.857861 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:09.857269    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:11.85725101 +0000 UTC m=+46.174779801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : configmap references non-existent config key: service-ca.crt
Jul 23 22:23:09.857861 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:09.857667    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 22:23:09.857861 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:09.857719    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert podName:36d92e8d-d0d3-4b4a-aa52-f67a77ccea41 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:11.857704213 +0000 UTC m=+46.175233017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tfzqx" (UID: "36d92e8d-d0d3-4b4a-aa52-f67a77ccea41") : secret "networking-console-plugin-cert" not found
Jul 23 22:23:09.857861 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:09.857776    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 22:23:09.857861 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:09.857810    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:11.857798238 +0000 UTC m=+46.175327028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : secret "router-metrics-certs-default" not found
Jul 23 22:23:10.509809 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:10.509721    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-85wbm" event={"ID":"5ae231e2-f5bf-4933-ad63-9c81b4931ce2","Type":"ContainerStarted","Data":"be1c30b08856694f2674b64643869d2756ee4c54ea114d3eb819301ccef2226c"}
Jul 23 22:23:10.666359 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:10.666269    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwlkg_68c30bcd-6fb8-4d32-adcc-881a10cdc158/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:10.772011 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:10.771869    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kwlkg_68c30bcd-6fb8-4d32-adcc-881a10cdc158/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:11.673393 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:11.673358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rrqfl\" (UID: \"3bb5c7c0-a03e-48b6-b948-993b360abcb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:11.673751 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:11.673468    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 23 22:23:11.673751 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:11.673522    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls podName:3bb5c7c0-a03e-48b6-b948-993b360abcb3 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:15.673509028 +0000 UTC m=+49.991037816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rrqfl" (UID: "3bb5c7c0-a03e-48b6-b948-993b360abcb3") : secret "samples-operator-tls" not found
Jul 23 22:23:11.773840 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:11.773800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:11.773992 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:11.773936    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:11.774033 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:11.774001    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls podName:031164ea-f2ff-4883-80bb-30979f504bb6 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:15.773985047 +0000 UTC m=+50.091513835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-h2t79" (UID: "031164ea-f2ff-4883-80bb-30979f504bb6") : secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:11.875172 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:11.875130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:11.875172 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:11.875177    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:11.875392 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:11.875217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:11.875392 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:11.875313    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 22:23:11.875392 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:11.875357    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 22:23:11.875542 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:11.875401    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert podName:36d92e8d-d0d3-4b4a-aa52-f67a77ccea41 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:15.875379457 +0000 UTC m=+50.192908248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tfzqx" (UID: "36d92e8d-d0d3-4b4a-aa52-f67a77ccea41") : secret "networking-console-plugin-cert" not found
Jul 23 22:23:11.875542 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:11.875421    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:15.875411728 +0000 UTC m=+50.192940516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : secret "router-metrics-certs-default" not found
Jul 23 22:23:11.875542 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:11.875438    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:15.875429088 +0000 UTC m=+50.192957875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : configmap references non-existent config key: service-ca.crt
Jul 23 22:23:15.525523 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.525245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh" event={"ID":"1e2ff89d-860c-44a9-b40b-983d106daea3","Type":"ContainerStarted","Data":"a6c2305be9004d9ca5ef478036097077c5067e53619d9a4d115cd4ae35ad41dd"}
Jul 23 22:23:15.527300 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.527185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vv74w" event={"ID":"bfba70cc-edef-4788-acd6-258f2a4a15c6","Type":"ContainerStarted","Data":"167813f040e7fc78cd9c7ddeac7c0e3e620f117566a2fd2694519e0df1d1bdf5"}
Jul 23 22:23:15.529936 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.528685    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf" event={"ID":"400dc06d-48bd-4183-afca-09e259ca6d48","Type":"ContainerStarted","Data":"ad17304ce6cd188939ce23e17757b6560297e31ad884d259bd4c7378503f5fc0"}
Jul 23 22:23:15.531426 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.531406    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/0.log"
Jul 23 22:23:15.531527 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.531442    2583 generic.go:356] "Generic (PLEG): container finished" podID="b29adb35-eeca-46dd-b18e-8a3cb48d4769" containerID="17e7f36233f8590e1f06717fc7ea69dd5a1911bc441f4a0ec9b24c9fcdee8928" exitCode=255
Jul 23 22:23:15.531589 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.531528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g" event={"ID":"b29adb35-eeca-46dd-b18e-8a3cb48d4769","Type":"ContainerDied","Data":"17e7f36233f8590e1f06717fc7ea69dd5a1911bc441f4a0ec9b24c9fcdee8928"}
Jul 23 22:23:15.531710 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.531694    2583 scope.go:117] "RemoveContainer" containerID="17e7f36233f8590e1f06717fc7ea69dd5a1911bc441f4a0ec9b24c9fcdee8928"
Jul 23 22:23:15.533249 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.533228    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-gbh2j" event={"ID":"4b018493-e500-4c1e-9b5c-88285f6a9dc4","Type":"ContainerStarted","Data":"fc7b2189814be50f4e292760046f7b08d98ee3b4fc54464305d839eed2f09f08"}
Jul 23 22:23:15.544975 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.544896    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh" podStartSLOduration=2.213447224 podStartE2EDuration="8.544878862s" totalImagesPullingTime="6.331431638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:15.54448323 +0000 UTC m=+49.862012053" watchObservedRunningTime="2026-07-23 22:23:15.544878862 +0000 UTC m=+49.862407672"
Jul 23 22:23:15.546466 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.546424    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-85wbm" podStartSLOduration=6.816683918 podStartE2EDuration="8.546408631s" totalImagesPullingTime="1.729724713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:10.544344995 +0000 UTC m=+44.861873805" watchObservedRunningTime="2026-07-23 22:23:15.546408631 +0000 UTC m=+49.863937442"
Jul 23 22:23:15.561842 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.561791    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-vv74w" podStartSLOduration=2.202082168 podStartE2EDuration="8.561776323s" totalImagesPullingTime="6.359694155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:15.561071611 +0000 UTC m=+49.878600422" watchObservedRunningTime="2026-07-23 22:23:15.561776323 +0000 UTC m=+49.879305133"
Jul 23 22:23:15.601363 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.601287    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf" podStartSLOduration=2.292014889 podStartE2EDuration="8.60126963s" totalImagesPullingTime="6.309254741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:15.579857489 +0000 UTC m=+49.897386299" watchObservedRunningTime="2026-07-23 22:23:15.60126963 +0000 UTC m=+49.918798439"
Jul 23 22:23:15.620947 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.620885    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-gbh2j" podStartSLOduration=2.181841734 podStartE2EDuration="8.620866704s" totalImagesPullingTime="6.43902497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:15.620112412 +0000 UTC m=+49.937641226" watchObservedRunningTime="2026-07-23 22:23:15.620866704 +0000 UTC m=+49.938395515"
Jul 23 22:23:15.708785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.708753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rrqfl\" (UID: \"3bb5c7c0-a03e-48b6-b948-993b360abcb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:15.708967 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:15.708945    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 23 22:23:15.709036 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:15.709024    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls podName:3bb5c7c0-a03e-48b6-b948-993b360abcb3 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:23.709003288 +0000 UTC m=+58.026532078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rrqfl" (UID: "3bb5c7c0-a03e-48b6-b948-993b360abcb3") : secret "samples-operator-tls" not found
Jul 23 22:23:15.809407 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.809378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:15.809563 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:15.809537    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:15.809623 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:15.809613    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls podName:031164ea-f2ff-4883-80bb-30979f504bb6 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:23.809592336 +0000 UTC m=+58.127121128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-h2t79" (UID: "031164ea-f2ff-4883-80bb-30979f504bb6") : secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:15.910460 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.910424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:15.910633 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.910494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:15.910633 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:15.910581    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 22:23:15.910806 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:15.910629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:15.910806 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:15.910650    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:23.910631599 +0000 UTC m=+58.228160402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : secret "router-metrics-certs-default" not found
Jul 23 22:23:15.910806 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:15.910706    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 22:23:15.910806 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:15.910733    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:23.910712778 +0000 UTC m=+58.228241579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : configmap references non-existent config key: service-ca.crt
Jul 23 22:23:15.910806 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:15.910761    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert podName:36d92e8d-d0d3-4b4a-aa52-f67a77ccea41 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:23.910751805 +0000 UTC m=+58.228280597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tfzqx" (UID: "36d92e8d-d0d3-4b4a-aa52-f67a77ccea41") : secret "networking-console-plugin-cert" not found
Jul 23 22:23:16.065209 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.065129    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7mbjs"]
Jul 23 22:23:16.086720 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.086693    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7mbjs"]
Jul 23 22:23:16.087033 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.086973    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.089983 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.089961    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h4r9r\""
Jul 23 22:23:16.090107 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.089998    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 22:23:16.090240 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.090227    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 22:23:16.112517 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.112494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfj2z\" (UniqueName: \"kubernetes.io/projected/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-api-access-bfj2z\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.112613 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.112530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4e6016d-f94a-41a4-bbc1-4303d8158985-data-volume\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.112613 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.112559    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.112733 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.112718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.112772 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.112758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4e6016d-f94a-41a4-bbc1-4303d8158985-crio-socket\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.213765 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.213731    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.213765 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.213771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4e6016d-f94a-41a4-bbc1-4303d8158985-crio-socket\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.214003 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.213823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfj2z\" (UniqueName: \"kubernetes.io/projected/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-api-access-bfj2z\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.214003 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.213840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4e6016d-f94a-41a4-bbc1-4303d8158985-data-volume\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.214003 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.213856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.214003 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.213857    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 22:23:16.214003 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.213995    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 22:23:16.214253 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.214045    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-rbac-proxy-cm podName:c4e6016d-f94a-41a4-bbc1-4303d8158985 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:16.714025314 +0000 UTC m=+51.031554101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7mbjs" (UID: "c4e6016d-f94a-41a4-bbc1-4303d8158985") : configmap "kube-rbac-proxy" not found
Jul 23 22:23:16.214253 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.214064    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls podName:c4e6016d-f94a-41a4-bbc1-4303d8158985 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:16.714054077 +0000 UTC m=+51.031582870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7mbjs" (UID: "c4e6016d-f94a-41a4-bbc1-4303d8158985") : secret "insights-runtime-extractor-tls" not found
Jul 23 22:23:16.214253 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.214189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4e6016d-f94a-41a4-bbc1-4303d8158985-crio-socket\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.214253 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.214200    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4e6016d-f94a-41a4-bbc1-4303d8158985-data-volume\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.230340 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.230297    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfj2z\" (UniqueName: \"kubernetes.io/projected/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-api-access-bfj2z\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.539245 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.539161    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/1.log"
Jul 23 22:23:16.539680 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.539603    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/0.log"
Jul 23 22:23:16.539680 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.539640    2583 generic.go:356] "Generic (PLEG): container finished" podID="b29adb35-eeca-46dd-b18e-8a3cb48d4769" containerID="6a553594939d11012227f7af92269c57bddddf27e18510219efe6d143a63edf3" exitCode=255
Jul 23 22:23:16.539783 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.539728    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g" event={"ID":"b29adb35-eeca-46dd-b18e-8a3cb48d4769","Type":"ContainerDied","Data":"6a553594939d11012227f7af92269c57bddddf27e18510219efe6d143a63edf3"}
Jul 23 22:23:16.539836 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.539812    2583 scope.go:117] "RemoveContainer" containerID="17e7f36233f8590e1f06717fc7ea69dd5a1911bc441f4a0ec9b24c9fcdee8928"
Jul 23 22:23:16.539994 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.539974    2583 scope.go:117] "RemoveContainer" containerID="6a553594939d11012227f7af92269c57bddddf27e18510219efe6d143a63edf3"
Jul 23 22:23:16.540210 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.540186    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-tnv5g_openshift-console-operator(b29adb35-eeca-46dd-b18e-8a3cb48d4769)\"" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g" podUID="b29adb35-eeca-46dd-b18e-8a3cb48d4769"
Jul 23 22:23:16.719745 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.719703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.719914 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.719815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:16.719914 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.719848    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 22:23:16.719914 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.719909    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-rbac-proxy-cm podName:c4e6016d-f94a-41a4-bbc1-4303d8158985 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:17.719895287 +0000 UTC m=+52.037424074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7mbjs" (UID: "c4e6016d-f94a-41a4-bbc1-4303d8158985") : configmap "kube-rbac-proxy" not found
Jul 23 22:23:16.720097 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.719950    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 22:23:16.720097 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.720029    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls podName:c4e6016d-f94a-41a4-bbc1-4303d8158985 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:17.720011261 +0000 UTC m=+52.037540068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7mbjs" (UID: "c4e6016d-f94a-41a4-bbc1-4303d8158985") : secret "insights-runtime-extractor-tls" not found
Jul 23 22:23:16.921554 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.921524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:16.921763 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:16.921594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:16.921763 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.921681    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 22:23:16.921763 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.921756    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls podName:3a3df64e-97c8-4a17-8848-6fd2c8fd4057 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:32.921741725 +0000 UTC m=+67.239270512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls") pod "dns-default-lnlzn" (UID: "3a3df64e-97c8-4a17-8848-6fd2c8fd4057") : secret "dns-default-metrics-tls" not found
Jul 23 22:23:16.921872 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.921687    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 22:23:16.921872 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.921784    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575b6566d-rrhk9: secret "image-registry-tls" not found
Jul 23 22:23:16.921872 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:16.921828    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls podName:60e8cde0-a29c-4982-9833-b31d8e864630 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:32.92181752 +0000 UTC m=+67.239346306 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls") pod "image-registry-575b6566d-rrhk9" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630") : secret "image-registry-tls" not found
Jul 23 22:23:17.060583 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.060555    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-68765_89ffe12b-74bf-4582-8395-b47766839015/dns-node-resolver/0.log"
Jul 23 22:23:17.178413 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.178343    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt"]
Jul 23 22:23:17.204579 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.204555    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt"
Jul 23 22:23:17.208062 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.208042    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 23 22:23:17.208183 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.208152    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 23 22:23:17.208238 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.208159    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2lgc7\""
Jul 23 22:23:17.219387 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.219363    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt"]
Jul 23 22:23:17.223737 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.223716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhxzs\" (UniqueName: \"kubernetes.io/projected/3e6516e0-bcc1-4cf1-ac64-2b1f0179c9f5-kube-api-access-qhxzs\") pod \"migrator-7cc9c7ffd7-xgwvt\" (UID: \"3e6516e0-bcc1-4cf1-ac64-2b1f0179c9f5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt"
Jul 23 22:23:17.324805 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.324783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxzs\" (UniqueName: \"kubernetes.io/projected/3e6516e0-bcc1-4cf1-ac64-2b1f0179c9f5-kube-api-access-qhxzs\") pod \"migrator-7cc9c7ffd7-xgwvt\" (UID: \"3e6516e0-bcc1-4cf1-ac64-2b1f0179c9f5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt"
Jul 23 22:23:17.334043 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.334024    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxzs\" (UniqueName: \"kubernetes.io/projected/3e6516e0-bcc1-4cf1-ac64-2b1f0179c9f5-kube-api-access-qhxzs\") pod \"migrator-7cc9c7ffd7-xgwvt\" (UID: \"3e6516e0-bcc1-4cf1-ac64-2b1f0179c9f5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt"
Jul 23 22:23:17.514192 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.514106    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt"
Jul 23 22:23:17.544091 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.544060    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/1.log"
Jul 23 22:23:17.544517 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.544502    2583 scope.go:117] "RemoveContainer" containerID="6a553594939d11012227f7af92269c57bddddf27e18510219efe6d143a63edf3"
Jul 23 22:23:17.544732 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:17.544706    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-tnv5g_openshift-console-operator(b29adb35-eeca-46dd-b18e-8a3cb48d4769)\"" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g" podUID="b29adb35-eeca-46dd-b18e-8a3cb48d4769"
Jul 23 22:23:17.643563 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.643446    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt"]
Jul 23 22:23:17.656364 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:17.656335    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6516e0_bcc1_4cf1_ac64_2b1f0179c9f5.slice/crio-ccc694e34e3eecd048333256874db8f54642e7cb66f6c515e6e8402d51f11d60 WatchSource:0}: Error finding container ccc694e34e3eecd048333256874db8f54642e7cb66f6c515e6e8402d51f11d60: Status 404 returned error can't find the container with id ccc694e34e3eecd048333256874db8f54642e7cb66f6c515e6e8402d51f11d60
Jul 23 22:23:17.729194 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.729171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:17.729338 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:17.729317    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 22:23:17.729382 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.729343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:17.729382 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:17.729377    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls podName:c4e6016d-f94a-41a4-bbc1-4303d8158985 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:19.729362213 +0000 UTC m=+54.046891004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7mbjs" (UID: "c4e6016d-f94a-41a4-bbc1-4303d8158985") : secret "insights-runtime-extractor-tls" not found
Jul 23 22:23:17.729810 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:17.729793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e6016d-f94a-41a4-bbc1-4303d8158985-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:18.042220 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.042188    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4xjxj_889cb48e-740e-4093-9908-67f62a225840/node-ca/0.log"
Jul 23 22:23:18.308019 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.307979    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:18.308196 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.308049    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:18.549186 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.549143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt" event={"ID":"3e6516e0-bcc1-4cf1-ac64-2b1f0179c9f5","Type":"ContainerStarted","Data":"ccc694e34e3eecd048333256874db8f54642e7cb66f6c515e6e8402d51f11d60"}
Jul 23 22:23:18.549655 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.549493    2583 scope.go:117] "RemoveContainer" containerID="6a553594939d11012227f7af92269c57bddddf27e18510219efe6d143a63edf3"
Jul 23 22:23:18.549718 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:18.549685    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-tnv5g_openshift-console-operator(b29adb35-eeca-46dd-b18e-8a3cb48d4769)\"" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g" podUID="b29adb35-eeca-46dd-b18e-8a3cb48d4769"
Jul 23 22:23:18.725547 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.725473    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-h52t4"]
Jul 23 22:23:18.737157 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.737050    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-h52t4"
Jul 23 22:23:18.738234 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.738213    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-h52t4"]
Jul 23 22:23:18.739844 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.739823    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 23 22:23:18.739965 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.739829    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 23 22:23:18.740039 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.739977    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 23 22:23:18.741090 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.741054    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dbhsq\""
Jul 23 22:23:18.741235 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.741216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 23 22:23:18.839444 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.839407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60e37dba-8c09-4c72-b632-6a61ab7792bd-signing-cabundle\") pod \"service-ca-58598695f7-h52t4\" (UID: \"60e37dba-8c09-4c72-b632-6a61ab7792bd\") " pod="openshift-service-ca/service-ca-58598695f7-h52t4"
Jul 23 22:23:18.839584 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.839499    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td29r\" (UniqueName: \"kubernetes.io/projected/60e37dba-8c09-4c72-b632-6a61ab7792bd-kube-api-access-td29r\") pod \"service-ca-58598695f7-h52t4\" (UID: \"60e37dba-8c09-4c72-b632-6a61ab7792bd\") " pod="openshift-service-ca/service-ca-58598695f7-h52t4"
Jul 23 22:23:18.839637 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.839591    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60e37dba-8c09-4c72-b632-6a61ab7792bd-signing-key\") pod \"service-ca-58598695f7-h52t4\" (UID: \"60e37dba-8c09-4c72-b632-6a61ab7792bd\") " pod="openshift-service-ca/service-ca-58598695f7-h52t4"
Jul 23 22:23:18.843038 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.843013    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-78sqp_72439e60-4ee5-4235-bd03-68c54f40aa4e/serve-healthcheck-canary/0.log"
Jul 23 22:23:18.940908 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.940874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60e37dba-8c09-4c72-b632-6a61ab7792bd-signing-cabundle\") pod \"service-ca-58598695f7-h52t4\" (UID: \"60e37dba-8c09-4c72-b632-6a61ab7792bd\") " pod="openshift-service-ca/service-ca-58598695f7-h52t4"
Jul 23 22:23:18.941087 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.940974    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td29r\" (UniqueName: \"kubernetes.io/projected/60e37dba-8c09-4c72-b632-6a61ab7792bd-kube-api-access-td29r\") pod \"service-ca-58598695f7-h52t4\" (UID: \"60e37dba-8c09-4c72-b632-6a61ab7792bd\") " pod="openshift-service-ca/service-ca-58598695f7-h52t4"
Jul 23 22:23:18.941087 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.941042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60e37dba-8c09-4c72-b632-6a61ab7792bd-signing-key\") pod \"service-ca-58598695f7-h52t4\" (UID: \"60e37dba-8c09-4c72-b632-6a61ab7792bd\") " pod="openshift-service-ca/service-ca-58598695f7-h52t4"
Jul 23 22:23:18.941508 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.941487    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60e37dba-8c09-4c72-b632-6a61ab7792bd-signing-cabundle\") pod \"service-ca-58598695f7-h52t4\" (UID: \"60e37dba-8c09-4c72-b632-6a61ab7792bd\") " pod="openshift-service-ca/service-ca-58598695f7-h52t4"
Jul 23 22:23:18.943490 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.943468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60e37dba-8c09-4c72-b632-6a61ab7792bd-signing-key\") pod \"service-ca-58598695f7-h52t4\" (UID: \"60e37dba-8c09-4c72-b632-6a61ab7792bd\") " pod="openshift-service-ca/service-ca-58598695f7-h52t4"
Jul 23 22:23:18.950326 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:18.950304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td29r\" (UniqueName: \"kubernetes.io/projected/60e37dba-8c09-4c72-b632-6a61ab7792bd-kube-api-access-td29r\") pod \"service-ca-58598695f7-h52t4\" (UID: \"60e37dba-8c09-4c72-b632-6a61ab7792bd\") " pod="openshift-service-ca/service-ca-58598695f7-h52t4"
Jul 23 22:23:19.047862 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:19.047783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-h52t4"
Jul 23 22:23:19.213105 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:19.212958    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-h52t4"]
Jul 23 22:23:19.215749 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:19.215724    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e37dba_8c09_4c72_b632_6a61ab7792bd.slice/crio-a70d0476b81144618f70dc5e0ea95c79e19f7bde7ea8a6564a6423b331a1918e WatchSource:0}: Error finding container a70d0476b81144618f70dc5e0ea95c79e19f7bde7ea8a6564a6423b331a1918e: Status 404 returned error can't find the container with id a70d0476b81144618f70dc5e0ea95c79e19f7bde7ea8a6564a6423b331a1918e
Jul 23 22:23:19.554989 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:19.554882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt" event={"ID":"3e6516e0-bcc1-4cf1-ac64-2b1f0179c9f5","Type":"ContainerStarted","Data":"e0354210b8fe09c5c519132ec73e4001ca9a7a954d1fc63fb59769cfefc46401"}
Jul 23 22:23:19.554989 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:19.554957    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt" event={"ID":"3e6516e0-bcc1-4cf1-ac64-2b1f0179c9f5","Type":"ContainerStarted","Data":"e62cafded80d45527121a6c52a44446906b92654344439d54cc8498bbe0f6fbc"}
Jul 23 22:23:19.556375 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:19.556349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-h52t4" event={"ID":"60e37dba-8c09-4c72-b632-6a61ab7792bd","Type":"ContainerStarted","Data":"f171315afdfbfc1c52634a0afe86e4a05f1bf1aaddb1a0fafff8602933db6074"}
Jul 23 22:23:19.556495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:19.556391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-h52t4" event={"ID":"60e37dba-8c09-4c72-b632-6a61ab7792bd","Type":"ContainerStarted","Data":"a70d0476b81144618f70dc5e0ea95c79e19f7bde7ea8a6564a6423b331a1918e"}
Jul 23 22:23:19.576059 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:19.576009    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xgwvt" podStartSLOduration=1.137829653 podStartE2EDuration="2.575989951s" totalImagesPullingTime="1.438160298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:19.575342781 +0000 UTC m=+53.892871589" watchObservedRunningTime="2026-07-23 22:23:19.575989951 +0000 UTC m=+53.893518761"
Jul 23 22:23:19.598368 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:19.598314    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-h52t4" podStartSLOduration=1.5983040320000002 podStartE2EDuration="1.598304032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:19.59786423 +0000 UTC m=+53.915393039" watchObservedRunningTime="2026-07-23 22:23:19.598304032 +0000 UTC m=+53.915832842"
Jul 23 22:23:19.749693 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:19.749657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:19.749820 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:19.749777    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 22:23:19.749860 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:19.749828    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls podName:c4e6016d-f94a-41a4-bbc1-4303d8158985 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:23.749815318 +0000 UTC m=+58.067344105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7mbjs" (UID: "c4e6016d-f94a-41a4-bbc1-4303d8158985") : secret "insights-runtime-extractor-tls" not found
Jul 23 22:23:23.786487 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:23.786448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:23.787049 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:23.786637    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 22:23:23.787049 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:23.786700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rrqfl\" (UID: \"3bb5c7c0-a03e-48b6-b948-993b360abcb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:23.787049 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:23.786723    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls podName:c4e6016d-f94a-41a4-bbc1-4303d8158985 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:31.786701334 +0000 UTC m=+66.104230137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7mbjs" (UID: "c4e6016d-f94a-41a4-bbc1-4303d8158985") : secret "insights-runtime-extractor-tls" not found
Jul 23 22:23:23.789002 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:23.788980    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bb5c7c0-a03e-48b6-b948-993b360abcb3-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rrqfl\" (UID: \"3bb5c7c0-a03e-48b6-b948-993b360abcb3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:23.818964 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:23.818919    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"
Jul 23 22:23:23.887967 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:23.887786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:23.887967 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:23.887896    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:23.887967 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:23.887971    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls podName:031164ea-f2ff-4883-80bb-30979f504bb6 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:39.887957154 +0000 UTC m=+74.205485942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-h2t79" (UID: "031164ea-f2ff-4883-80bb-30979f504bb6") : secret "cluster-monitoring-operator-tls" not found
Jul 23 22:23:23.944429 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:23.944396    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl"]
Jul 23 22:23:23.988327 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:23.988300    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:23.988422 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:23.988347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:23.988422 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:23.988381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:23.988493 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:23.988441    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 22:23:23.988525 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:23.988494    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert podName:36d92e8d-d0d3-4b4a-aa52-f67a77ccea41 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:39.988479307 +0000 UTC m=+74.306008098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tfzqx" (UID: "36d92e8d-d0d3-4b4a-aa52-f67a77ccea41") : secret "networking-console-plugin-cert" not found
Jul 23 22:23:23.988525 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:23.988514    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 22:23:23.988591 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:23.988539    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:39.988528716 +0000 UTC m=+74.306057503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : configmap references non-existent config key: service-ca.crt
Jul 23 22:23:23.988591 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:23.988554    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs podName:f88b4c99-967b-4ded-8e2d-00bf67380e09 nodeName:}" failed. No retries permitted until 2026-07-23 22:23:39.988548063 +0000 UTC m=+74.306076850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs") pod "router-default-84db8994fd-csdvv" (UID: "f88b4c99-967b-4ded-8e2d-00bf67380e09") : secret "router-metrics-certs-default" not found
Jul 23 22:23:24.576825 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:24.576784    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl" event={"ID":"3bb5c7c0-a03e-48b6-b948-993b360abcb3","Type":"ContainerStarted","Data":"229a7cd3c04c65fc89c0c4f219eaa5cf38796d1cb9acb98af296fe727ff6dc16"}
Jul 23 22:23:25.582165 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:25.582131    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl" event={"ID":"3bb5c7c0-a03e-48b6-b948-993b360abcb3","Type":"ContainerStarted","Data":"c175246a45228aabbc486b70c8feab922fb53af4408b7d7fe1bc96f81b6cf5a9"}
Jul 23 22:23:26.586575 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:26.586533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl" event={"ID":"3bb5c7c0-a03e-48b6-b948-993b360abcb3","Type":"ContainerStarted","Data":"5cf86ce5c8b0d3557cc1d4af9681b889e374472f89428d7b9d39f2e6e904bca5"}
Jul 23 22:23:26.603733 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:26.603685    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rrqfl" podStartSLOduration=18.07674398 podStartE2EDuration="19.603670322s" totalImagesPullingTime="1.526926342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:26.602908358 +0000 UTC m=+60.920437169" watchObservedRunningTime="2026-07-23 22:23:26.603670322 +0000 UTC m=+60.921199108"
Jul 23 22:23:27.804918 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:27.804889    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92v68"
Jul 23 22:23:30.223903 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:30.223866    2583 scope.go:117] "RemoveContainer" containerID="6a553594939d11012227f7af92269c57bddddf27e18510219efe6d143a63edf3"
Jul 23 22:23:30.600835 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:30.600809    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:23:30.601204 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:30.601189    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/1.log"
Jul 23 22:23:30.601267 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:30.601221    2583 generic.go:356] "Generic (PLEG): container finished" podID="b29adb35-eeca-46dd-b18e-8a3cb48d4769" containerID="2a02918b93aa511324953485aa869010e162f93354d1d95803cbe656e1d8626d" exitCode=255
Jul 23 22:23:30.601267 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:30.601260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g" event={"ID":"b29adb35-eeca-46dd-b18e-8a3cb48d4769","Type":"ContainerDied","Data":"2a02918b93aa511324953485aa869010e162f93354d1d95803cbe656e1d8626d"}
Jul 23 22:23:30.601367 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:30.601308    2583 scope.go:117] "RemoveContainer" containerID="6a553594939d11012227f7af92269c57bddddf27e18510219efe6d143a63edf3"
Jul 23 22:23:30.601643 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:30.601625    2583 scope.go:117] "RemoveContainer" containerID="2a02918b93aa511324953485aa869010e162f93354d1d95803cbe656e1d8626d"
Jul 23 22:23:30.601817 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:30.601796    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-tnv5g_openshift-console-operator(b29adb35-eeca-46dd-b18e-8a3cb48d4769)\"" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g" podUID="b29adb35-eeca-46dd-b18e-8a3cb48d4769"
Jul 23 22:23:31.606295 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:31.606264    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:23:31.863888 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:31.863794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:31.866104 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:31.866085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e6016d-f94a-41a4-bbc1-4303d8158985-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mbjs\" (UID: \"c4e6016d-f94a-41a4-bbc1-4303d8158985\") " pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:31.964250 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:31.964201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:23:31.964250 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:31.964253    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lg5\" (UniqueName: \"kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5\") pod \"network-check-target-tvd42\" (UID: \"492cf2fb-60f4-494f-8608-72d8f00b071f\") " pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:23:31.966697 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:31.966664    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6lg5\" (UniqueName: \"kubernetes.io/projected/492cf2fb-60f4-494f-8608-72d8f00b071f-kube-api-access-x6lg5\") pod \"network-check-target-tvd42\" (UID: \"492cf2fb-60f4-494f-8608-72d8f00b071f\") " pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:23:31.967365 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:31.967348    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 22:23:31.976769 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:31.976751    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/729340b2-7e6e-474a-8475-db80f5a1ac8b-original-pull-secret\") pod \"global-pull-secret-syncer-mtsxj\" (UID: \"729340b2-7e6e-474a-8475-db80f5a1ac8b\") " pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:23:31.999676 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:31.999657    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h4r9r\""
Jul 23 22:23:32.006966 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.006945    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7mbjs"
Jul 23 22:23:32.065345 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.065314    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:23:32.068969 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.068944    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 22:23:32.077615 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.077594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d068205-8a0a-4ae6-b8a2-f831bf0e9752-metrics-certs\") pod \"network-metrics-daemon-d6b9c\" (UID: \"6d068205-8a0a-4ae6-b8a2-f831bf0e9752\") " pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:23:32.130813 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.130742    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7mbjs"]
Jul 23 22:23:32.132527 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.132509    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtsxj"
Jul 23 22:23:32.133762 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:32.133735    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e6016d_f94a_41a4_bbc1_4303d8158985.slice/crio-8104c17efb6de23b91a95ddf830709e31faed6b115a83176558a590fe7ab53e4 WatchSource:0}: Error finding container 8104c17efb6de23b91a95ddf830709e31faed6b115a83176558a590fe7ab53e4: Status 404 returned error can't find the container with id 8104c17efb6de23b91a95ddf830709e31faed6b115a83176558a590fe7ab53e4
Jul 23 22:23:32.141362 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.141344    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-655rt\""
Jul 23 22:23:32.144823 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.144804    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2pbtn\""
Jul 23 22:23:32.149155 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.149137    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:23:32.153111 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.153092    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6b9c"
Jul 23 22:23:32.275409 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.275301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mtsxj"]
Jul 23 22:23:32.283276 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:32.283249    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729340b2_7e6e_474a_8475_db80f5a1ac8b.slice/crio-d71271bbba1532c8601968efb10963780e24816530a704d17b230f5c10ff8bea WatchSource:0}: Error finding container d71271bbba1532c8601968efb10963780e24816530a704d17b230f5c10ff8bea: Status 404 returned error can't find the container with id d71271bbba1532c8601968efb10963780e24816530a704d17b230f5c10ff8bea
Jul 23 22:23:32.294181 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.294153    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d6b9c"]
Jul 23 22:23:32.297328 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:32.297299    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d068205_8a0a_4ae6_b8a2_f831bf0e9752.slice/crio-55e52a33973d2c28c2170db190f953f24f37c78f241eb5164a18a555550a106f WatchSource:0}: Error finding container 55e52a33973d2c28c2170db190f953f24f37c78f241eb5164a18a555550a106f: Status 404 returned error can't find the container with id 55e52a33973d2c28c2170db190f953f24f37c78f241eb5164a18a555550a106f
Jul 23 22:23:32.310446 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.310390    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tvd42"]
Jul 23 22:23:32.312753 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:32.312733    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492cf2fb_60f4_494f_8608_72d8f00b071f.slice/crio-fe7d4f1e999c0612399b04ebc33a35d15a664dbab5c96b4c3fdc2f82f734ebbb WatchSource:0}: Error finding container fe7d4f1e999c0612399b04ebc33a35d15a664dbab5c96b4c3fdc2f82f734ebbb: Status 404 returned error can't find the container with id fe7d4f1e999c0612399b04ebc33a35d15a664dbab5c96b4c3fdc2f82f734ebbb
Jul 23 22:23:32.611382 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.611343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d6b9c" event={"ID":"6d068205-8a0a-4ae6-b8a2-f831bf0e9752","Type":"ContainerStarted","Data":"55e52a33973d2c28c2170db190f953f24f37c78f241eb5164a18a555550a106f"}
Jul 23 22:23:32.612583 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.612556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mtsxj" event={"ID":"729340b2-7e6e-474a-8475-db80f5a1ac8b","Type":"ContainerStarted","Data":"d71271bbba1532c8601968efb10963780e24816530a704d17b230f5c10ff8bea"}
Jul 23 22:23:32.614462 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.614430    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tvd42" event={"ID":"492cf2fb-60f4-494f-8608-72d8f00b071f","Type":"ContainerStarted","Data":"144ffe118b56ade637eed2fbca03843ffaa250f71417acd0dd047e2353410ca4"}
Jul 23 22:23:32.614552 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.614473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tvd42" event={"ID":"492cf2fb-60f4-494f-8608-72d8f00b071f","Type":"ContainerStarted","Data":"fe7d4f1e999c0612399b04ebc33a35d15a664dbab5c96b4c3fdc2f82f734ebbb"}
Jul 23 22:23:32.614607 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.614553    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:23:32.615851 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.615814    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7mbjs" event={"ID":"c4e6016d-f94a-41a4-bbc1-4303d8158985","Type":"ContainerStarted","Data":"8104c17efb6de23b91a95ddf830709e31faed6b115a83176558a590fe7ab53e4"}
Jul 23 22:23:32.633555 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.633513    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tvd42" podStartSLOduration=66.633498779 podStartE2EDuration="1m6.633498779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:32.63218384 +0000 UTC m=+66.949712650" watchObservedRunningTime="2026-07-23 22:23:32.633498779 +0000 UTC m=+66.951027587"
Jul 23 22:23:32.975811 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.975176    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:32.975811 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.975258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:32.979227 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.979128    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls\") pod \"image-registry-575b6566d-rrhk9\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") " pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:32.980127 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:32.980103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a3df64e-97c8-4a17-8848-6fd2c8fd4057-metrics-tls\") pod \"dns-default-lnlzn\" (UID: \"3a3df64e-97c8-4a17-8848-6fd2c8fd4057\") " pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:33.124212 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:33.124128    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dk6d4\""
Jul 23 22:23:33.131436 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:33.131404    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:33.149613 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:33.149591    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fmr8p\""
Jul 23 22:23:33.157913 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:33.157890    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:33.649636 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:33.649552    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lnlzn"]
Jul 23 22:23:33.652671 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:33.652637    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3df64e_97c8_4a17_8848_6fd2c8fd4057.slice/crio-c4234b9156c7ddca3b41fc22ae9b7799fc07ad4341510b340fa0e46b84c284ec WatchSource:0}: Error finding container c4234b9156c7ddca3b41fc22ae9b7799fc07ad4341510b340fa0e46b84c284ec: Status 404 returned error can't find the container with id c4234b9156c7ddca3b41fc22ae9b7799fc07ad4341510b340fa0e46b84c284ec
Jul 23 22:23:33.675747 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:33.675718    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-575b6566d-rrhk9"]
Jul 23 22:23:33.841964 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:33.841907    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e8cde0_a29c_4982_9833_b31d8e864630.slice/crio-0718e7afc32c67f88d657b9effeb8f0c405b12c53cf6a3c5d482e2655b56fee0 WatchSource:0}: Error finding container 0718e7afc32c67f88d657b9effeb8f0c405b12c53cf6a3c5d482e2655b56fee0: Status 404 returned error can't find the container with id 0718e7afc32c67f88d657b9effeb8f0c405b12c53cf6a3c5d482e2655b56fee0
Jul 23 22:23:34.631805 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:34.630858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575b6566d-rrhk9" event={"ID":"60e8cde0-a29c-4982-9833-b31d8e864630","Type":"ContainerStarted","Data":"6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720"}
Jul 23 22:23:34.631805 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:34.630959    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575b6566d-rrhk9" event={"ID":"60e8cde0-a29c-4982-9833-b31d8e864630","Type":"ContainerStarted","Data":"0718e7afc32c67f88d657b9effeb8f0c405b12c53cf6a3c5d482e2655b56fee0"}
Jul 23 22:23:34.631805 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:34.631735    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:34.634839 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:34.634318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d6b9c" event={"ID":"6d068205-8a0a-4ae6-b8a2-f831bf0e9752","Type":"ContainerStarted","Data":"f8664b2bfaf697593fa870cbfc7e96fafe5cb9a54f239a46652531c088380014"}
Jul 23 22:23:34.634839 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:34.634366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d6b9c" event={"ID":"6d068205-8a0a-4ae6-b8a2-f831bf0e9752","Type":"ContainerStarted","Data":"81f11fb57ae43490c961613c02c286f58772af83629bec3f3f76723d1f92b701"}
Jul 23 22:23:34.637302 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:34.637243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnlzn" event={"ID":"3a3df64e-97c8-4a17-8848-6fd2c8fd4057","Type":"ContainerStarted","Data":"c4234b9156c7ddca3b41fc22ae9b7799fc07ad4341510b340fa0e46b84c284ec"}
Jul 23 22:23:34.662679 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:34.662126    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-575b6566d-rrhk9" podStartSLOduration=68.662108918 podStartE2EDuration="1m8.662108918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:34.661354739 +0000 UTC m=+68.978883549" watchObservedRunningTime="2026-07-23 22:23:34.662108918 +0000 UTC m=+68.979637728"
Jul 23 22:23:34.695725 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:34.695644    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d6b9c" podStartSLOduration=67.124030068 podStartE2EDuration="1m8.695624651s" totalImagesPullingTime="1.571594583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:34.693791717 +0000 UTC m=+69.011320527" watchObservedRunningTime="2026-07-23 22:23:34.695624651 +0000 UTC m=+69.013153461"
Jul 23 22:23:38.308996 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:38.308960    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:38.309489 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:38.309022    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:38.309489 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:38.309430    2583 scope.go:117] "RemoveContainer" containerID="2a02918b93aa511324953485aa869010e162f93354d1d95803cbe656e1d8626d"
Jul 23 22:23:38.309639 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:23:38.309618    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-tnv5g_openshift-console-operator(b29adb35-eeca-46dd-b18e-8a3cb48d4769)\"" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g" podUID="b29adb35-eeca-46dd-b18e-8a3cb48d4769"
Jul 23 22:23:39.065612 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.065579    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6"]
Jul 23 22:23:39.070546 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.070521    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"]
Jul 23 22:23:39.070739 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.070682    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6"
Jul 23 22:23:39.074134 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.074110    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 23 22:23:39.074266 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.074154    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4x6vq\""
Jul 23 22:23:39.074919 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.074894    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:39.075848 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.075766    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 23 22:23:39.076006 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.075935    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 23 22:23:39.076006 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.076002    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 23 22:23:39.081725 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.079050    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 23 22:23:39.081725 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.081090    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"]
Jul 23 22:23:39.089464 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.089382    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6"]
Jul 23 22:23:39.089564 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.089483    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.092301 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.092274    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 23 22:23:39.092404 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.092308    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 23 22:23:39.092404 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.092370    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 23 22:23:39.092521 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.092423    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 23 22:23:39.095618 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.095322    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"]
Jul 23 22:23:39.106525 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.106453    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"]
Jul 23 22:23:39.236643 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.236502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58452f4-94e2-4827-9d02-76b45d0d37b0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.237041 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.236912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9bpv\" (UniqueName: \"kubernetes.io/projected/c302463e-41f5-489f-af94-906b12f8ced0-kube-api-access-s9bpv\") pod \"managed-serviceaccount-addon-agent-58b859594b-lsfx6\" (UID: \"c302463e-41f5-489f-af94-906b12f8ced0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6"
Jul 23 22:23:39.237041 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.236983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-hub\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.237209 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.237149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.237209 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.237195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l88lk\" (UniqueName: \"kubernetes.io/projected/f58452f4-94e2-4827-9d02-76b45d0d37b0-kube-api-access-l88lk\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.237317 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.237246    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c302463e-41f5-489f-af94-906b12f8ced0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58b859594b-lsfx6\" (UID: \"c302463e-41f5-489f-af94-906b12f8ced0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6"
Jul 23 22:23:39.237317 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.237279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.237394 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.237367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftjb\" (UniqueName: \"kubernetes.io/projected/e54acf91-2023-43df-a94b-785423c495ae-kube-api-access-zftjb\") pod \"klusterlet-addon-workmgr-8454d896d-8w785\" (UID: \"e54acf91-2023-43df-a94b-785423c495ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:39.237442 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.237416    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e54acf91-2023-43df-a94b-785423c495ae-klusterlet-config\") pod \"klusterlet-addon-workmgr-8454d896d-8w785\" (UID: \"e54acf91-2023-43df-a94b-785423c495ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:39.237479 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.237471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e54acf91-2023-43df-a94b-785423c495ae-tmp\") pod \"klusterlet-addon-workmgr-8454d896d-8w785\" (UID: \"e54acf91-2023-43df-a94b-785423c495ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:39.237528 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.237493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-ca\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.338829 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.338798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c302463e-41f5-489f-af94-906b12f8ced0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58b859594b-lsfx6\" (UID: \"c302463e-41f5-489f-af94-906b12f8ced0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6"
Jul 23 22:23:39.339281 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.338842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.339281 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.338875    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftjb\" (UniqueName: \"kubernetes.io/projected/e54acf91-2023-43df-a94b-785423c495ae-kube-api-access-zftjb\") pod \"klusterlet-addon-workmgr-8454d896d-8w785\" (UID: \"e54acf91-2023-43df-a94b-785423c495ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:39.339281 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.338919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e54acf91-2023-43df-a94b-785423c495ae-klusterlet-config\") pod \"klusterlet-addon-workmgr-8454d896d-8w785\" (UID: \"e54acf91-2023-43df-a94b-785423c495ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:39.339281 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.338987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e54acf91-2023-43df-a94b-785423c495ae-tmp\") pod \"klusterlet-addon-workmgr-8454d896d-8w785\" (UID: \"e54acf91-2023-43df-a94b-785423c495ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:39.339281 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.339010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-ca\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.339537 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.339400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58452f4-94e2-4827-9d02-76b45d0d37b0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.339537 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.339435    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e54acf91-2023-43df-a94b-785423c495ae-tmp\") pod \"klusterlet-addon-workmgr-8454d896d-8w785\" (UID: \"e54acf91-2023-43df-a94b-785423c495ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:39.339537 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.339490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9bpv\" (UniqueName: \"kubernetes.io/projected/c302463e-41f5-489f-af94-906b12f8ced0-kube-api-access-s9bpv\") pod \"managed-serviceaccount-addon-agent-58b859594b-lsfx6\" (UID: \"c302463e-41f5-489f-af94-906b12f8ced0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6"
Jul 23 22:23:39.339537 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.339526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-hub\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.339730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.339558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.339730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.339600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l88lk\" (UniqueName: \"kubernetes.io/projected/f58452f4-94e2-4827-9d02-76b45d0d37b0-kube-api-access-l88lk\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.340232 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.340206    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58452f4-94e2-4827-9d02-76b45d0d37b0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.342020 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.341974    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-ca\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.342389 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.342350    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e54acf91-2023-43df-a94b-785423c495ae-klusterlet-config\") pod \"klusterlet-addon-workmgr-8454d896d-8w785\" (UID: \"e54acf91-2023-43df-a94b-785423c495ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:39.342561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.342538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.342863 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.342832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.343057 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.343034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c302463e-41f5-489f-af94-906b12f8ced0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58b859594b-lsfx6\" (UID: \"c302463e-41f5-489f-af94-906b12f8ced0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6"
Jul 23 22:23:39.343185 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.343167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f58452f4-94e2-4827-9d02-76b45d0d37b0-hub\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.357117 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.357074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftjb\" (UniqueName: \"kubernetes.io/projected/e54acf91-2023-43df-a94b-785423c495ae-kube-api-access-zftjb\") pod \"klusterlet-addon-workmgr-8454d896d-8w785\" (UID: \"e54acf91-2023-43df-a94b-785423c495ae\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:39.357117 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.357083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l88lk\" (UniqueName: \"kubernetes.io/projected/f58452f4-94e2-4827-9d02-76b45d0d37b0-kube-api-access-l88lk\") pod \"cluster-proxy-proxy-agent-5f67758f74-qg2ch\" (UID: \"f58452f4-94e2-4827-9d02-76b45d0d37b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.364120 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.364098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9bpv\" (UniqueName: \"kubernetes.io/projected/c302463e-41f5-489f-af94-906b12f8ced0-kube-api-access-s9bpv\") pod \"managed-serviceaccount-addon-agent-58b859594b-lsfx6\" (UID: \"c302463e-41f5-489f-af94-906b12f8ced0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6"
Jul 23 22:23:39.395441 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.395416    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6"
Jul 23 22:23:39.404870 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.404850    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:39.411779 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.411753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"
Jul 23 22:23:39.805638 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.805589    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"]
Jul 23 22:23:39.811390 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:39.811360    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54acf91_2023_43df_a94b_785423c495ae.slice/crio-e810e6b8f5cb3cf21a5cf8ad16d599b48e795a3c9fd35ea3de8ea416087d32cb WatchSource:0}: Error finding container e810e6b8f5cb3cf21a5cf8ad16d599b48e795a3c9fd35ea3de8ea416087d32cb: Status 404 returned error can't find the container with id e810e6b8f5cb3cf21a5cf8ad16d599b48e795a3c9fd35ea3de8ea416087d32cb
Jul 23 22:23:39.835813 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.835677    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch"]
Jul 23 22:23:39.848844 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.848699    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6"]
Jul 23 22:23:39.853527 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:39.853482    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc302463e_41f5_489f_af94_906b12f8ced0.slice/crio-0257070f2bfd99537351525b969ad1b618dc86e56bbd515897ed4ef54873f0a5 WatchSource:0}: Error finding container 0257070f2bfd99537351525b969ad1b618dc86e56bbd515897ed4ef54873f0a5: Status 404 returned error can't find the container with id 0257070f2bfd99537351525b969ad1b618dc86e56bbd515897ed4ef54873f0a5
Jul 23 22:23:39.945639 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.945608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:39.948290 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:39.948265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/031164ea-f2ff-4883-80bb-30979f504bb6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-h2t79\" (UID: \"031164ea-f2ff-4883-80bb-30979f504bb6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:40.046170 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.046134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:40.046344 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.046199    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:40.046344 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.046247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:40.046823 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.046803    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88b4c99-967b-4ded-8e2d-00bf67380e09-service-ca-bundle\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:40.048650 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.048620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f88b4c99-967b-4ded-8e2d-00bf67380e09-metrics-certs\") pod \"router-default-84db8994fd-csdvv\" (UID: \"f88b4c99-967b-4ded-8e2d-00bf67380e09\") " pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:40.048736 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.048620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36d92e8d-d0d3-4b4a-aa52-f67a77ccea41-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tfzqx\" (UID: \"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:40.068161 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.068142    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-44gg6\""
Jul 23 22:23:40.076154 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.076130    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"
Jul 23 22:23:40.175089 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.175067    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-h52z8\""
Jul 23 22:23:40.182936 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.182891    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:40.200729 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.200711    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4lp4n\""
Jul 23 22:23:40.208828 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.208799    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"
Jul 23 22:23:40.221695 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.221675    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79"]
Jul 23 22:23:40.225593 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:40.225563    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031164ea_f2ff_4883_80bb_30979f504bb6.slice/crio-4fcf9786e8a9f576f6de0c209d8e3c2b9bf22f1eeafd7d6c810664730466fd0a WatchSource:0}: Error finding container 4fcf9786e8a9f576f6de0c209d8e3c2b9bf22f1eeafd7d6c810664730466fd0a: Status 404 returned error can't find the container with id 4fcf9786e8a9f576f6de0c209d8e3c2b9bf22f1eeafd7d6c810664730466fd0a
Jul 23 22:23:40.362559 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.362528    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84db8994fd-csdvv"]
Jul 23 22:23:40.367790 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:40.367756    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88b4c99_967b_4ded_8e2d_00bf67380e09.slice/crio-495b1f15b90668c290d9c6b88e213240ee67005f4f79e3809a9c1351c107e778 WatchSource:0}: Error finding container 495b1f15b90668c290d9c6b88e213240ee67005f4f79e3809a9c1351c107e778: Status 404 returned error can't find the container with id 495b1f15b90668c290d9c6b88e213240ee67005f4f79e3809a9c1351c107e778
Jul 23 22:23:40.405385 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.405349    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx"]
Jul 23 22:23:40.412837 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:40.412805    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d92e8d_d0d3_4b4a_aa52_f67a77ccea41.slice/crio-1b58e77bdf8a2e515fd09713750fc7afb5994a3e5eae0daf46a4b2b1acfa1c2a WatchSource:0}: Error finding container 1b58e77bdf8a2e515fd09713750fc7afb5994a3e5eae0daf46a4b2b1acfa1c2a: Status 404 returned error can't find the container with id 1b58e77bdf8a2e515fd09713750fc7afb5994a3e5eae0daf46a4b2b1acfa1c2a
Jul 23 22:23:40.665308 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.665218    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6" event={"ID":"c302463e-41f5-489f-af94-906b12f8ced0","Type":"ContainerStarted","Data":"0257070f2bfd99537351525b969ad1b618dc86e56bbd515897ed4ef54873f0a5"}
Jul 23 22:23:40.669989 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.669914    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mtsxj" event={"ID":"729340b2-7e6e-474a-8475-db80f5a1ac8b","Type":"ContainerStarted","Data":"40e68e3bf76a41f0c7293bc34c76bcf69e29287645580825fa8210a783684596"}
Jul 23 22:23:40.674826 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.674264    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84db8994fd-csdvv" event={"ID":"f88b4c99-967b-4ded-8e2d-00bf67380e09","Type":"ContainerStarted","Data":"3b9042d23712127b7819d3f58990931f2a4ac29cdd50b6ca97bbf5ce9df5af25"}
Jul 23 22:23:40.674826 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.674309    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84db8994fd-csdvv" event={"ID":"f88b4c99-967b-4ded-8e2d-00bf67380e09","Type":"ContainerStarted","Data":"495b1f15b90668c290d9c6b88e213240ee67005f4f79e3809a9c1351c107e778"}
Jul 23 22:23:40.677504 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.677427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785" event={"ID":"e54acf91-2023-43df-a94b-785423c495ae","Type":"ContainerStarted","Data":"e810e6b8f5cb3cf21a5cf8ad16d599b48e795a3c9fd35ea3de8ea416087d32cb"}
Jul 23 22:23:40.682463 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.682387    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch" event={"ID":"f58452f4-94e2-4827-9d02-76b45d0d37b0","Type":"ContainerStarted","Data":"aaa118293d56bab9af3be6811943c92101b463e8a06031cb6dd577235db2b35b"}
Jul 23 22:23:40.684346 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.684320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79" event={"ID":"031164ea-f2ff-4883-80bb-30979f504bb6","Type":"ContainerStarted","Data":"4fcf9786e8a9f576f6de0c209d8e3c2b9bf22f1eeafd7d6c810664730466fd0a"}
Jul 23 22:23:40.687022 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.686966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx" event={"ID":"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41","Type":"ContainerStarted","Data":"1b58e77bdf8a2e515fd09713750fc7afb5994a3e5eae0daf46a4b2b1acfa1c2a"}
Jul 23 22:23:40.694137 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.693741    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7mbjs" event={"ID":"c4e6016d-f94a-41a4-bbc1-4303d8158985","Type":"ContainerStarted","Data":"fb57d95abe1e4e0861331e7db025236992fbf3b8c47feccf0a888ccde38614f8"}
Jul 23 22:23:40.697155 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.696298    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mtsxj" podStartSLOduration=67.346975191 podStartE2EDuration="1m14.696281411s" totalImagesPullingTime="7.34930622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:40.693963411 +0000 UTC m=+75.011492223" watchObservedRunningTime="2026-07-23 22:23:40.696281411 +0000 UTC m=+75.013810243"
Jul 23 22:23:40.700067 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.699328    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnlzn" event={"ID":"3a3df64e-97c8-4a17-8848-6fd2c8fd4057","Type":"ContainerStarted","Data":"81da4ee3619bf0e543a408f7d3136032878db38c5304235b1af8eebe1a4a2cd7"}
Jul 23 22:23:40.700067 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.699377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnlzn" event={"ID":"3a3df64e-97c8-4a17-8848-6fd2c8fd4057","Type":"ContainerStarted","Data":"3faee1ccf342bfdae9a2ddb6a24141e71655fd05d5053c13f1530235ad82ead2"}
Jul 23 22:23:40.700067 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.699949    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:40.762960 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.762849    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lnlzn" podStartSLOduration=33.783043347 podStartE2EDuration="39.762815738s" totalImagesPullingTime="5.979772391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:40.761602742 +0000 UTC m=+75.079131552" watchObservedRunningTime="2026-07-23 22:23:40.762815738 +0000 UTC m=+75.080344539"
Jul 23 22:23:40.763308 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:40.763271    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-84db8994fd-csdvv" podStartSLOduration=33.763255874 podStartE2EDuration="33.763255874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:40.733010114 +0000 UTC m=+75.050538924" watchObservedRunningTime="2026-07-23 22:23:40.763255874 +0000 UTC m=+75.080784684"
Jul 23 22:23:41.183582 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:41.183504    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:41.186727 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:41.186491    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:41.713187 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:41.713115    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:41.714428 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:41.714000    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-84db8994fd-csdvv"
Jul 23 22:23:50.756209 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.756164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785" event={"ID":"e54acf91-2023-43df-a94b-785423c495ae","Type":"ContainerStarted","Data":"29fe04f641720a2605181d204dd1e9c3f3770994c560327a9a65a908fffde9fb"}
Jul 23 22:23:50.756729 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.756484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:50.757955 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.757913    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785"
Jul 23 22:23:50.758839 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.758813    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch" event={"ID":"f58452f4-94e2-4827-9d02-76b45d0d37b0","Type":"ContainerStarted","Data":"0ae3f83be7f4d5385da03842e5c54018273875031a72a178d69c1b565bd495d9"}
Jul 23 22:23:50.758989 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.758846    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch" event={"ID":"f58452f4-94e2-4827-9d02-76b45d0d37b0","Type":"ContainerStarted","Data":"a0a2ab5518a033ac2c79f7a0eae228174f75f44a29a034b08949927eb63bdcc3"}
Jul 23 22:23:50.758989 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.758865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch" event={"ID":"f58452f4-94e2-4827-9d02-76b45d0d37b0","Type":"ContainerStarted","Data":"21aca89b67143e7f6831d85003fb45f2e9de10de9d85b1b5c82ea995e4e0b87b"}
Jul 23 22:23:50.760555 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.760527    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79" event={"ID":"031164ea-f2ff-4883-80bb-30979f504bb6","Type":"ContainerStarted","Data":"c2618e31ec03550d2444daffdf5acfb8662b87fae6dd6bdf57c0ee8e1ad8889b"}
Jul 23 22:23:50.762325 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.762272    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx" event={"ID":"36d92e8d-d0d3-4b4a-aa52-f67a77ccea41","Type":"ContainerStarted","Data":"57994a96785f3fcaaaccdaca835225c344d52341d1e9e00a5b5dd88079a8000b"}
Jul 23 22:23:50.764352 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.764327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7mbjs" event={"ID":"c4e6016d-f94a-41a4-bbc1-4303d8158985","Type":"ContainerStarted","Data":"6b49731526c9f103a609053fdf9a7d882267344efee41d3276a5a3b47e5fa428"}
Jul 23 22:23:50.765787 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.765767    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6" event={"ID":"c302463e-41f5-489f-af94-906b12f8ced0","Type":"ContainerStarted","Data":"f4f9f01b5e1521e9218794a16060588110ced03303978bf246bc4bb61d6ee32d"}
Jul 23 22:23:50.776468 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.776418    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8454d896d-8w785" podStartSLOduration=1.895391115 podStartE2EDuration="11.776405774s" totalImagesPullingTime="9.881014659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:50.77481168 +0000 UTC m=+85.092340516" watchObservedRunningTime="2026-07-23 22:23:50.776405774 +0000 UTC m=+85.093934582"
Jul 23 22:23:50.793984 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.793943    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-h2t79" podStartSLOduration=34.325593681 podStartE2EDuration="43.793916925s" totalImagesPullingTime="9.468323244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:50.792664624 +0000 UTC m=+85.110193434" watchObservedRunningTime="2026-07-23 22:23:50.793916925 +0000 UTC m=+85.111445733"
Jul 23 22:23:50.809740 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.809694    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tfzqx" podStartSLOduration=33.531363261 podStartE2EDuration="42.80967267s" totalImagesPullingTime="9.278309409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:50.809572389 +0000 UTC m=+85.127101197" watchObservedRunningTime="2026-07-23 22:23:50.80967267 +0000 UTC m=+85.127201479"
Jul 23 22:23:50.834052 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.834003    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f67758f74-qg2ch" podStartSLOduration=1.985368568 podStartE2EDuration="11.833988162s" totalImagesPullingTime="9.848619594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:50.831230549 +0000 UTC m=+85.148759359" watchObservedRunningTime="2026-07-23 22:23:50.833988162 +0000 UTC m=+85.151516970"
Jul 23 22:23:50.867689 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:50.867643    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b859594b-lsfx6" podStartSLOduration=2.027571623 podStartE2EDuration="11.86762797s" totalImagesPullingTime="9.840056347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:50.866848056 +0000 UTC m=+85.184376867" watchObservedRunningTime="2026-07-23 22:23:50.86762797 +0000 UTC m=+85.185156781"
Jul 23 22:23:51.720533 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:51.720467    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lnlzn"
Jul 23 22:23:52.223976 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:52.223934    2583 scope.go:117] "RemoveContainer" containerID="2a02918b93aa511324953485aa869010e162f93354d1d95803cbe656e1d8626d"
Jul 23 22:23:53.137460 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:53.137421    2583 patch_prober.go:28] interesting pod/image-registry-575b6566d-rrhk9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 23 22:23:53.137656 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:53.137484    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-575b6566d-rrhk9" podUID="60e8cde0-a29c-4982-9833-b31d8e864630" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 22:23:53.782555 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:53.782472    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:23:53.783043 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:53.782564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g" event={"ID":"b29adb35-eeca-46dd-b18e-8a3cb48d4769","Type":"ContainerStarted","Data":"b8f5d293b1018f1a34fbad6f38da3e4b6adb5f1e1bff1a54bdbc0f759ff5ffa1"}
Jul 23 22:23:53.783043 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:53.782965    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:53.785219 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:53.785188    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7mbjs" event={"ID":"c4e6016d-f94a-41a4-bbc1-4303d8158985","Type":"ContainerStarted","Data":"a4b7ca8dfc4517f5b14bbf5eee19492449e397d5f9b327d5a240db8223c3bc71"}
Jul 23 22:23:53.808667 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:53.808617    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g" podStartSLOduration=40.435043947 podStartE2EDuration="46.808603282s" totalImagesPullingTime="6.373559335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:53.806766693 +0000 UTC m=+88.124295501" watchObservedRunningTime="2026-07-23 22:23:53.808603282 +0000 UTC m=+88.126132090"
Jul 23 22:23:53.827283 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:53.827228    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7mbjs" podStartSLOduration=16.80531467 podStartE2EDuration="37.827213409s" totalImagesPullingTime="21.021898739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:23:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:23:53.826383672 +0000 UTC m=+88.143912482" watchObservedRunningTime="2026-07-23 22:23:53.827213409 +0000 UTC m=+88.144742217"
Jul 23 22:23:54.015589 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:54.015555    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-tnv5g"
Jul 23 22:23:56.650482 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:56.650450    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:23:58.848985 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.848949    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vv2b5"]
Jul 23 22:23:58.855395 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.855367    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:58.860836 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.860806    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 22:23:58.860984 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.860835    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 22:23:58.861058 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.860883    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 22:23:58.861222 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.860884    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9zrvx\""
Jul 23 22:23:58.861531 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.860890    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 22:23:58.911799 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.911764    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-wtmp\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:58.911988 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.911815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-accelerators-collector-config\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:58.911988 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.911908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-textfile\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:58.911988 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.911972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d81ce0b1-47e7-43ea-b8bb-da8395263376-root\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:58.912147 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.912021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d81ce0b1-47e7-43ea-b8bb-da8395263376-sys\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:58.912147 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.912067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-tls\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:58.912147 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.912095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:58.912256 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.912161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d81ce0b1-47e7-43ea-b8bb-da8395263376-metrics-client-ca\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:58.912256 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:58.912216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ndw9\" (UniqueName: \"kubernetes.io/projected/d81ce0b1-47e7-43ea-b8bb-da8395263376-kube-api-access-4ndw9\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.012633 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ndw9\" (UniqueName: \"kubernetes.io/projected/d81ce0b1-47e7-43ea-b8bb-da8395263376-kube-api-access-4ndw9\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.012633 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012635    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-wtmp\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.012873 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-accelerators-collector-config\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.012873 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-textfile\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.012873 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d81ce0b1-47e7-43ea-b8bb-da8395263376-root\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.012873 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012776    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d81ce0b1-47e7-43ea-b8bb-da8395263376-root\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.012873 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012807    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-wtmp\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.012873 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d81ce0b1-47e7-43ea-b8bb-da8395263376-sys\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.013177 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-tls\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.013177 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.013177 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012970    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d81ce0b1-47e7-43ea-b8bb-da8395263376-sys\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.013177 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.012990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d81ce0b1-47e7-43ea-b8bb-da8395263376-metrics-client-ca\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.013177 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.013090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-textfile\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.013396 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.013378    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-accelerators-collector-config\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.013519 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.013489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d81ce0b1-47e7-43ea-b8bb-da8395263376-metrics-client-ca\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.015278 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.015255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.015278 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.015270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d81ce0b1-47e7-43ea-b8bb-da8395263376-node-exporter-tls\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.023219 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.023195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ndw9\" (UniqueName: \"kubernetes.io/projected/d81ce0b1-47e7-43ea-b8bb-da8395263376-kube-api-access-4ndw9\") pod \"node-exporter-vv2b5\" (UID: \"d81ce0b1-47e7-43ea-b8bb-da8395263376\") " pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.165632 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.165539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vv2b5"
Jul 23 22:23:59.178689 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:23:59.178660    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81ce0b1_47e7_43ea_b8bb_da8395263376.slice/crio-4b5b6f6d5687bfe708d7272cee226d4a01ef7da1e5f7b1b078e774860dcfe1cb WatchSource:0}: Error finding container 4b5b6f6d5687bfe708d7272cee226d4a01ef7da1e5f7b1b078e774860dcfe1cb: Status 404 returned error can't find the container with id 4b5b6f6d5687bfe708d7272cee226d4a01ef7da1e5f7b1b078e774860dcfe1cb
Jul 23 22:23:59.812098 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:23:59.812062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vv2b5" event={"ID":"d81ce0b1-47e7-43ea-b8bb-da8395263376","Type":"ContainerStarted","Data":"4b5b6f6d5687bfe708d7272cee226d4a01ef7da1e5f7b1b078e774860dcfe1cb"}
Jul 23 22:24:00.817601 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:00.817569    2583 generic.go:356] "Generic (PLEG): container finished" podID="d81ce0b1-47e7-43ea-b8bb-da8395263376" containerID="1de99954dd5ae4b6e68a9e6b90faf988e9435b286ad2a5bb56ac57c31a489b66" exitCode=0
Jul 23 22:24:00.818024 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:00.817641    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vv2b5" event={"ID":"d81ce0b1-47e7-43ea-b8bb-da8395263376","Type":"ContainerDied","Data":"1de99954dd5ae4b6e68a9e6b90faf988e9435b286ad2a5bb56ac57c31a489b66"}
Jul 23 22:24:01.823843 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:01.823809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vv2b5" event={"ID":"d81ce0b1-47e7-43ea-b8bb-da8395263376","Type":"ContainerStarted","Data":"df5c31c6a594e45711fc3b2779a633042fce22d11e52bd269278b2bcc86731f6"}
Jul 23 22:24:01.824252 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:01.823891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vv2b5" event={"ID":"d81ce0b1-47e7-43ea-b8bb-da8395263376","Type":"ContainerStarted","Data":"ec4789f58dc12f5bfae54c666b10bc645b24279e624011eb2b3cd47cf091b10f"}
Jul 23 22:24:01.848789 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:01.848719    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vv2b5" podStartSLOduration=2.797339923 podStartE2EDuration="3.848702111s" totalImagesPullingTime="981.162995ms" totalInitContainerRuntime="70.199193ms" isStatefulPod=true podCreationTimestamp="2026-07-23 22:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:24:01.846726451 +0000 UTC m=+96.164255261" watchObservedRunningTime="2026-07-23 22:24:01.848702111 +0000 UTC m=+96.166230919"
Jul 23 22:24:02.131179 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:02.131100    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-575b6566d-rrhk9"]
Jul 23 22:24:03.624473 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:03.624444    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tvd42"
Jul 23 22:24:27.151286 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.151243    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-575b6566d-rrhk9" podUID="60e8cde0-a29c-4982-9833-b31d8e864630" containerName="registry" containerID="cri-o://6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720" gracePeriod=30
Jul 23 22:24:27.394020 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.393993    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:24:27.561647 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.561602    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-bound-sa-token\") pod \"60e8cde0-a29c-4982-9833-b31d8e864630\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") "
Jul 23 22:24:27.561828 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.561660    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-installation-pull-secrets\") pod \"60e8cde0-a29c-4982-9833-b31d8e864630\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") "
Jul 23 22:24:27.561828 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.561691    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60e8cde0-a29c-4982-9833-b31d8e864630-ca-trust-extracted\") pod \"60e8cde0-a29c-4982-9833-b31d8e864630\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") "
Jul 23 22:24:27.561828 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.561784    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl7l9\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-kube-api-access-zl7l9\") pod \"60e8cde0-a29c-4982-9833-b31d8e864630\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") "
Jul 23 22:24:27.561828 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.561818    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-image-registry-private-configuration\") pod \"60e8cde0-a29c-4982-9833-b31d8e864630\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") "
Jul 23 22:24:27.562054 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.561847    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-trusted-ca\") pod \"60e8cde0-a29c-4982-9833-b31d8e864630\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") "
Jul 23 22:24:27.562054 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.561878    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-registry-certificates\") pod \"60e8cde0-a29c-4982-9833-b31d8e864630\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") "
Jul 23 22:24:27.562054 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.561963    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls\") pod \"60e8cde0-a29c-4982-9833-b31d8e864630\" (UID: \"60e8cde0-a29c-4982-9833-b31d8e864630\") "
Jul 23 22:24:27.562316 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.562289    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "60e8cde0-a29c-4982-9833-b31d8e864630" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:24:27.562780 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.562740    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "60e8cde0-a29c-4982-9833-b31d8e864630" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:24:27.564340 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.564293    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "60e8cde0-a29c-4982-9833-b31d8e864630" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:24:27.564484 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.564458    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "60e8cde0-a29c-4982-9833-b31d8e864630" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:24:27.564604 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.564457    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-kube-api-access-zl7l9" (OuterVolumeSpecName: "kube-api-access-zl7l9") pod "60e8cde0-a29c-4982-9833-b31d8e864630" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630"). InnerVolumeSpecName "kube-api-access-zl7l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:24:27.564604 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.564555    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "60e8cde0-a29c-4982-9833-b31d8e864630" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:24:27.564717 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.564624    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "60e8cde0-a29c-4982-9833-b31d8e864630" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:24:27.569854 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.569829    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e8cde0-a29c-4982-9833-b31d8e864630-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "60e8cde0-a29c-4982-9833-b31d8e864630" (UID: "60e8cde0-a29c-4982-9833-b31d8e864630"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:24:27.663154 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.663109    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl7l9\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-kube-api-access-zl7l9\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:24:27.663154 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.663146    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-image-registry-private-configuration\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:24:27.663154 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.663158    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-trusted-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:24:27.663154 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.663167    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60e8cde0-a29c-4982-9833-b31d8e864630-registry-certificates\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:24:27.663427 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.663177    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-registry-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:24:27.663427 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.663185    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60e8cde0-a29c-4982-9833-b31d8e864630-bound-sa-token\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:24:27.663427 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.663192    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60e8cde0-a29c-4982-9833-b31d8e864630-installation-pull-secrets\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:24:27.663427 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.663200    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60e8cde0-a29c-4982-9833-b31d8e864630-ca-trust-extracted\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:24:27.921266 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.921177    2583 generic.go:356] "Generic (PLEG): container finished" podID="60e8cde0-a29c-4982-9833-b31d8e864630" containerID="6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720" exitCode=0
Jul 23 22:24:27.921266 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.921222    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575b6566d-rrhk9"
Jul 23 22:24:27.921435 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.921264    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575b6566d-rrhk9" event={"ID":"60e8cde0-a29c-4982-9833-b31d8e864630","Type":"ContainerDied","Data":"6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720"}
Jul 23 22:24:27.921435 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.921345    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575b6566d-rrhk9" event={"ID":"60e8cde0-a29c-4982-9833-b31d8e864630","Type":"ContainerDied","Data":"0718e7afc32c67f88d657b9effeb8f0c405b12c53cf6a3c5d482e2655b56fee0"}
Jul 23 22:24:27.921435 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.921372    2583 scope.go:117] "RemoveContainer" containerID="6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720"
Jul 23 22:24:27.930248 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.930229    2583 scope.go:117] "RemoveContainer" containerID="6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720"
Jul 23 22:24:27.930526 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:24:27.930505    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720\": container with ID starting with 6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720 not found: ID does not exist" containerID="6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720"
Jul 23 22:24:27.930594 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.930535    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720"} err="failed to get container status \"6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720\": rpc error: code = NotFound desc = could not find container \"6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720\": container with ID starting with 6fc47ca9179071a713e2ac4c9a8d04c222d4b837c0609ce90d988a6339463720 not found: ID does not exist"
Jul 23 22:24:27.959074 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.959048    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-575b6566d-rrhk9"]
Jul 23 22:24:27.963982 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:27.963960    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-575b6566d-rrhk9"]
Jul 23 22:24:28.228354 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:28.228263    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60e8cde0-a29c-4982-9833-b31d8e864630" path="/var/lib/kubelet/pods/60e8cde0-a29c-4982-9833-b31d8e864630/volumes"
Jul 23 22:24:30.934634 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:30.934598    2583 generic.go:356] "Generic (PLEG): container finished" podID="4b018493-e500-4c1e-9b5c-88285f6a9dc4" containerID="fc7b2189814be50f4e292760046f7b08d98ee3b4fc54464305d839eed2f09f08" exitCode=0
Jul 23 22:24:30.935062 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:30.934665    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-gbh2j" event={"ID":"4b018493-e500-4c1e-9b5c-88285f6a9dc4","Type":"ContainerDied","Data":"fc7b2189814be50f4e292760046f7b08d98ee3b4fc54464305d839eed2f09f08"}
Jul 23 22:24:30.935062 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:30.935031    2583 scope.go:117] "RemoveContainer" containerID="fc7b2189814be50f4e292760046f7b08d98ee3b4fc54464305d839eed2f09f08"
Jul 23 22:24:31.940304 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:31.940270    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-gbh2j" event={"ID":"4b018493-e500-4c1e-9b5c-88285f6a9dc4","Type":"ContainerStarted","Data":"7e533d1174985cd390552a06c1afd7b610f2424ad23682a7b8ce76095ebb1c4f"}
Jul 23 22:24:40.976570 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:40.976537    2583 generic.go:356] "Generic (PLEG): container finished" podID="400dc06d-48bd-4183-afca-09e259ca6d48" containerID="ad17304ce6cd188939ce23e17757b6560297e31ad884d259bd4c7378503f5fc0" exitCode=0
Jul 23 22:24:40.976981 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:40.976618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf" event={"ID":"400dc06d-48bd-4183-afca-09e259ca6d48","Type":"ContainerDied","Data":"ad17304ce6cd188939ce23e17757b6560297e31ad884d259bd4c7378503f5fc0"}
Jul 23 22:24:40.977052 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:40.977034    2583 scope.go:117] "RemoveContainer" containerID="ad17304ce6cd188939ce23e17757b6560297e31ad884d259bd4c7378503f5fc0"
Jul 23 22:24:41.982026 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:41.981994    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dtpzf" event={"ID":"400dc06d-48bd-4183-afca-09e259ca6d48","Type":"ContainerStarted","Data":"5e0804b1feaa62c46c325c50d4fe4688d6a803a02a8839a03d7cda5c31bb8ce0"}
Jul 23 22:24:47.007531 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:47.007494    2583 generic.go:356] "Generic (PLEG): container finished" podID="1e2ff89d-860c-44a9-b40b-983d106daea3" containerID="a6c2305be9004d9ca5ef478036097077c5067e53619d9a4d115cd4ae35ad41dd" exitCode=0
Jul 23 22:24:47.007943 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:47.007568    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh" event={"ID":"1e2ff89d-860c-44a9-b40b-983d106daea3","Type":"ContainerDied","Data":"a6c2305be9004d9ca5ef478036097077c5067e53619d9a4d115cd4ae35ad41dd"}
Jul 23 22:24:47.007943 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:47.007887    2583 scope.go:117] "RemoveContainer" containerID="a6c2305be9004d9ca5ef478036097077c5067e53619d9a4d115cd4ae35ad41dd"
Jul 23 22:24:48.012938 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:24:48.012901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-6pdgh" event={"ID":"1e2ff89d-860c-44a9-b40b-983d106daea3","Type":"ContainerStarted","Data":"e658c3bec33e9012258d4d4a5ea7349903cf664d9723ca5aa7487b1f3c74e2ee"}
Jul 23 22:27:09.710026 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.709990    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"]
Jul 23 22:27:09.710441 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.710289    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e8cde0-a29c-4982-9833-b31d8e864630" containerName="registry"
Jul 23 22:27:09.710441 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.710302    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e8cde0-a29c-4982-9833-b31d8e864630" containerName="registry"
Jul 23 22:27:09.710441 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.710372    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="60e8cde0-a29c-4982-9833-b31d8e864630" containerName="registry"
Jul 23 22:27:09.713285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.713268    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.720936 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.720896    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 22:27:09.721066 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.721047    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 22:27:09.725806 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.725789    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 22:27:09.726030 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.726015    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5vgr8\""
Jul 23 22:27:09.726654 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.726641    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 22:27:09.733248 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.733226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f6pb\" (UniqueName: \"kubernetes.io/projected/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-kube-api-access-6f6pb\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.733357 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.733268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-tmp\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.733357 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.733300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.733474 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.733381    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-manifests\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.733474 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.733417    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.738500 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.738478    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"]
Jul 23 22:27:09.834526 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.834492    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f6pb\" (UniqueName: \"kubernetes.io/projected/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-kube-api-access-6f6pb\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.834696 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.834538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-tmp\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.834696 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.834564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.834696 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.834586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-manifests\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.834696 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.834607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.835075 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.835033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-manifests\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.836975 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.836955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-tmp\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.837194 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.837173    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.837231 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.837191    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:09.844690 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:09.844666    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f6pb\" (UniqueName: \"kubernetes.io/projected/49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b-kube-api-access-6f6pb\") pod \"opendatahub-operator-controller-manager-fc7cc8c86-kzm29\" (UID: \"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:10.022509 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:10.022433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:10.154775 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:10.154751    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"]
Jul 23 22:27:10.157031 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:27:10.157002    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a5bb1e_bb5d_4188_84b4_bcb0f53a3c9b.slice/crio-150aaa10da0b7a5dfb033d16f72dd775d5b857f47ac8c7f189fcadffce4cc0d3 WatchSource:0}: Error finding container 150aaa10da0b7a5dfb033d16f72dd775d5b857f47ac8c7f189fcadffce4cc0d3: Status 404 returned error can't find the container with id 150aaa10da0b7a5dfb033d16f72dd775d5b857f47ac8c7f189fcadffce4cc0d3
Jul 23 22:27:10.538285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:10.538251    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29" event={"ID":"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b","Type":"ContainerStarted","Data":"150aaa10da0b7a5dfb033d16f72dd775d5b857f47ac8c7f189fcadffce4cc0d3"}
Jul 23 22:27:13.553638 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:13.553554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29" event={"ID":"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b","Type":"ContainerStarted","Data":"bc16b472252067d89e2d6356e837474a4dfe75f95f2c6fdf262b9e6bf63bc8f5"}
Jul 23 22:27:14.559206 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:14.559170    2583 generic.go:356] "Generic (PLEG): container finished" podID="49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b" containerID="bc16b472252067d89e2d6356e837474a4dfe75f95f2c6fdf262b9e6bf63bc8f5" exitCode=0
Jul 23 22:27:14.559563 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:14.559258    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29" event={"ID":"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b","Type":"ContainerDied","Data":"bc16b472252067d89e2d6356e837474a4dfe75f95f2c6fdf262b9e6bf63bc8f5"}
Jul 23 22:27:15.565588 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:15.565551    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29" event={"ID":"49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b","Type":"ContainerStarted","Data":"53cc1d7af3d0bc1d3da42a24f748af64a6e60ca71e03329059e571809690d1bf"}
Jul 23 22:27:15.566043 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:15.565696    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:15.595439 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:15.595385    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29" podStartSLOduration=2.059283042 podStartE2EDuration="6.59536807s" totalImagesPullingTime="3.529486663s" totalInitContainerRuntime="1.006598365s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:27:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:27:15.593814643 +0000 UTC m=+289.911343449" watchObservedRunningTime="2026-07-23 22:27:15.59536807 +0000 UTC m=+289.912896868"
Jul 23 22:27:26.115454 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:26.115421    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:27:26.122149 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:26.122125    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:27:26.125697 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:26.125672    2583 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 22:27:26.572005 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:26.571971    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-fc7cc8c86-kzm29"
Jul 23 22:27:36.854099 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.854063    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"]
Jul 23 22:27:36.858864 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.858845    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:36.861885 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.861864    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 22:27:36.862020 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.861920    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kz72d\""
Jul 23 22:27:36.862123 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.862110    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 23 22:27:36.862580 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.862562    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 23 22:27:36.862580 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.862577    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 23 22:27:36.872833 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.872796    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"]
Jul 23 22:27:36.944639 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.944611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:36.944785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.944641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:36.944785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.944667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ead33c0-068d-4bd6-9348-cadd6918c921-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:36.944785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.944730    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:36.944975 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.944790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:36.944975 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.944841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g58h\" (UniqueName: \"kubernetes.io/projected/7ead33c0-068d-4bd6-9348-cadd6918c921-kube-api-access-8g58h\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:36.944975 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:36.944879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7ead33c0-068d-4bd6-9348-cadd6918c921-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.046190 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.046154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g58h\" (UniqueName: \"kubernetes.io/projected/7ead33c0-068d-4bd6-9348-cadd6918c921-kube-api-access-8g58h\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.046190 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.046189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7ead33c0-068d-4bd6-9348-cadd6918c921-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.046423 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.046227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.046423 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.046243    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.046423 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.046264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ead33c0-068d-4bd6-9348-cadd6918c921-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.046577 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.046448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.046577 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.046495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.047131 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.047106    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.048940 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.048894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7ead33c0-068d-4bd6-9348-cadd6918c921-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.049196 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.049175    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.049245 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.049220    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.049402 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.049382    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ead33c0-068d-4bd6-9348-cadd6918c921-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.054680 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.054655    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ead33c0-068d-4bd6-9348-cadd6918c921-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.055738 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.055718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g58h\" (UniqueName: \"kubernetes.io/projected/7ead33c0-068d-4bd6-9348-cadd6918c921-kube-api-access-8g58h\") pod \"istiod-openshift-gateway-94bb8fbfd-wm4lz\" (UID: \"7ead33c0-068d-4bd6-9348-cadd6918c921\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.167872 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.167807    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:37.301471 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.301446    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"]
Jul 23 22:27:37.304100 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:27:37.304058    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ead33c0_068d_4bd6_9348_cadd6918c921.slice/crio-515f1ffa043282ede702f8681c759d717a9e3924b93a0c4b62c08d0e9c9d6600 WatchSource:0}: Error finding container 515f1ffa043282ede702f8681c759d717a9e3924b93a0c4b62c08d0e9c9d6600: Status 404 returned error can't find the container with id 515f1ffa043282ede702f8681c759d717a9e3924b93a0c4b62c08d0e9c9d6600
Jul 23 22:27:37.306045 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.306028    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 22:27:37.657165 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:37.657128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz" event={"ID":"7ead33c0-068d-4bd6-9348-cadd6918c921","Type":"ContainerStarted","Data":"515f1ffa043282ede702f8681c759d717a9e3924b93a0c4b62c08d0e9c9d6600"}
Jul 23 22:27:39.060365 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.060332    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"]
Jul 23 22:27:39.067263 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.067241    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.072070 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.072040    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 23 22:27:39.072070 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.072059    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 23 22:27:39.072280 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.072059    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 23 22:27:39.072280 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.072127    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 23 22:27:39.072280 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.072041    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 23 22:27:39.072595 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.072569    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rjgzs\""
Jul 23 22:27:39.078018 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.077998    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"]
Jul 23 22:27:39.166236 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.166202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-cert\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.166236 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.166242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-manager-config\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.166457 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.166278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kz2s\" (UniqueName: \"kubernetes.io/projected/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-kube-api-access-8kz2s\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.166457 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.166362    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-metrics-cert\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.267746 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.267708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-cert\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.267983 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.267761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-manager-config\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.267983 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.267803    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kz2s\" (UniqueName: \"kubernetes.io/projected/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-kube-api-access-8kz2s\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.267983 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.267847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-metrics-cert\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.268730 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.268697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-manager-config\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.270454 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.270432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-cert\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.270567 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.270503    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-metrics-cert\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.293101 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.293077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kz2s\" (UniqueName: \"kubernetes.io/projected/5916f2f0-298c-425d-9a2d-4c1e7b2fdebf-kube-api-access-8kz2s\") pod \"lws-controller-manager-58df94799c-gqwq4\" (UID: \"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf\") " pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.379099 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.379028    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:39.709446 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.709406    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 23 22:27:39.709544 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.709497    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 23 22:27:39.787445 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:39.787421    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"]
Jul 23 22:27:39.789191 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:27:39.789165    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5916f2f0_298c_425d_9a2d_4c1e7b2fdebf.slice/crio-3c090c968c99cac254ee4baab508a1b31154da4c2f045e5527d04f57b6d08a8c WatchSource:0}: Error finding container 3c090c968c99cac254ee4baab508a1b31154da4c2f045e5527d04f57b6d08a8c: Status 404 returned error can't find the container with id 3c090c968c99cac254ee4baab508a1b31154da4c2f045e5527d04f57b6d08a8c
Jul 23 22:27:40.675960 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:40.675852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4" event={"ID":"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf","Type":"ContainerStarted","Data":"3c090c968c99cac254ee4baab508a1b31154da4c2f045e5527d04f57b6d08a8c"}
Jul 23 22:27:40.682431 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:40.680118    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz" event={"ID":"7ead33c0-068d-4bd6-9348-cadd6918c921","Type":"ContainerStarted","Data":"4266513bae49adb9944f59b8e0da3dbcde844b240a18a638492faabee06aee73"}
Jul 23 22:27:40.682431 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:40.681722    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:40.683397 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:40.683128    2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-wm4lz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 22:27:40.683397 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:40.683177    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz" podUID="7ead33c0-068d-4bd6-9348-cadd6918c921" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 22:27:40.713958 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:40.712968    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz" podStartSLOduration=2.309900626 podStartE2EDuration="4.712899307s" totalImagesPullingTime="2.402998681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:27:40.711260542 +0000 UTC m=+315.028789347" watchObservedRunningTime="2026-07-23 22:27:40.712899307 +0000 UTC m=+315.030428117"
Jul 23 22:27:41.686728 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:41.686695    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wm4lz"
Jul 23 22:27:42.692146 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:42.692107    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4" event={"ID":"5916f2f0-298c-425d-9a2d-4c1e7b2fdebf","Type":"ContainerStarted","Data":"1b5eac8a5679420cfd5c2f94d34d4852b54c63e78e12befddeffd3f366760cca"}
Jul 23 22:27:42.692528 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:42.692293    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:27:42.712873 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:42.712826    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4" podStartSLOduration=1.2684009600000001 podStartE2EDuration="3.712812619s" totalImagesPullingTime="2.444411659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:27:42.710547406 +0000 UTC m=+317.028076237" watchObservedRunningTime="2026-07-23 22:27:42.712812619 +0000 UTC m=+317.030341427"
Jul 23 22:27:53.699457 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:27:53.699426    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58df94799c-gqwq4"
Jul 23 22:32:17.346101 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.346063    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"]
Jul 23 22:32:17.349345 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.349327    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.351976 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.351954    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 22:32:17.352093 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.351981    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 23 22:32:17.353188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.353169    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dxxl7\""
Jul 23 22:32:17.353280 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.353204    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 22:32:17.362734 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.362711    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"]
Jul 23 22:32:17.494960 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.494908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee20b09-3e01-4352-9c14-442ba6bc44f3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.495131 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.494971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzkmb\" (UniqueName: \"kubernetes.io/projected/9ee20b09-3e01-4352-9c14-442ba6bc44f3-kube-api-access-tzkmb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.495131 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.495035    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.495131 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.495069    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.495131 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.495093    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.495264 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.495180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.596447 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.596360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee20b09-3e01-4352-9c14-442ba6bc44f3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.596447 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.596408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkmb\" (UniqueName: \"kubernetes.io/projected/9ee20b09-3e01-4352-9c14-442ba6bc44f3-kube-api-access-tzkmb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.596447 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.596438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.596706 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.596462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.596706 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.596482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.596706 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.596517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.596992 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.596969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.596992 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.596984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.597120 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.597032    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.598720 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.598688    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee20b09-3e01-4352-9c14-442ba6bc44f3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.598917 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.598897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee20b09-3e01-4352-9c14-442ba6bc44f3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.604860 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.604836    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzkmb\" (UniqueName: \"kubernetes.io/projected/9ee20b09-3e01-4352-9c14-442ba6bc44f3-kube-api-access-tzkmb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh4db\" (UID: \"9ee20b09-3e01-4352-9c14-442ba6bc44f3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.660729 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.660677    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:17.793556 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.793528    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"]
Jul 23 22:32:17.796430 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:32:17.796403    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee20b09_3e01_4352_9c14_442ba6bc44f3.slice/crio-a46c8c18dfa98a56b6b555de9f6b735f7429420af9fd58c794de3a03e81ca1de WatchSource:0}: Error finding container a46c8c18dfa98a56b6b555de9f6b735f7429420af9fd58c794de3a03e81ca1de: Status 404 returned error can't find the container with id a46c8c18dfa98a56b6b555de9f6b735f7429420af9fd58c794de3a03e81ca1de
Jul 23 22:32:17.845065 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:17.845033    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db" event={"ID":"9ee20b09-3e01-4352-9c14-442ba6bc44f3","Type":"ContainerStarted","Data":"a46c8c18dfa98a56b6b555de9f6b735f7429420af9fd58c794de3a03e81ca1de"}
Jul 23 22:32:19.857819 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:19.857762    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db" event={"ID":"9ee20b09-3e01-4352-9c14-442ba6bc44f3","Type":"ContainerStarted","Data":"e420195efd61d7d0a4b361110bb385408ba8adeb439b214fe5726dbca5332d03"}
Jul 23 22:32:19.878432 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:19.878378    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db" podStartSLOduration=1.892767756 podStartE2EDuration="2.878362406s" totalImagesPullingTime="985.59465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:32:19.876291399 +0000 UTC m=+594.193820208" watchObservedRunningTime="2026-07-23 22:32:19.878362406 +0000 UTC m=+594.195891216"
Jul 23 22:32:26.144143 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:26.144115    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:32:26.153616 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:26.153591    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:32:27.661789 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:27.661737    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:27.661789 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:27.661808    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:27.674450 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:27.674419    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:27.902594 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:27.902559    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh4db"
Jul 23 22:32:35.795285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:35.795249    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"]
Jul 23 22:32:35.800685 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:35.800661    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:35.803432 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:35.803399    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 23 22:32:35.810493 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:35.810471    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"]
Jul 23 22:32:35.966246 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:35.966211    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:35.966246 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:35.966249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46kz7\" (UniqueName: \"kubernetes.io/projected/5445b100-83bb-4857-8f99-9c12589f86b5-kube-api-access-46kz7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:35.966500 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:35.966269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:35.966500 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:35.966292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:35.966500 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:35.966378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5445b100-83bb-4857-8f99-9c12589f86b5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:35.966500 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:35.966476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.067882 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.067843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5445b100-83bb-4857-8f99-9c12589f86b5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.068112 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.067961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.068112 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.068032    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.068112 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.068060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46kz7\" (UniqueName: \"kubernetes.io/projected/5445b100-83bb-4857-8f99-9c12589f86b5-kube-api-access-46kz7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.068112 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.068085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.068112 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.068108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.068467 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.068446    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.068545 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.068524    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.068629 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.068605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.070296 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.070273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5445b100-83bb-4857-8f99-9c12589f86b5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.070593 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.070573    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5445b100-83bb-4857-8f99-9c12589f86b5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.076167 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.076150    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46kz7\" (UniqueName: \"kubernetes.io/projected/5445b100-83bb-4857-8f99-9c12589f86b5-kube-api-access-46kz7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7\" (UID: \"5445b100-83bb-4857-8f99-9c12589f86b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.111539 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.111508    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:36.246303 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.246278    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"]
Jul 23 22:32:36.248564 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:32:36.248534    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5445b100_83bb_4857_8f99_9c12589f86b5.slice/crio-ecb104679f09c063c86d85dfc37aa8173746bf037c19f2baa0d6bbec0716aff9 WatchSource:0}: Error finding container ecb104679f09c063c86d85dfc37aa8173746bf037c19f2baa0d6bbec0716aff9: Status 404 returned error can't find the container with id ecb104679f09c063c86d85dfc37aa8173746bf037c19f2baa0d6bbec0716aff9
Jul 23 22:32:36.932379 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.932341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7" event={"ID":"5445b100-83bb-4857-8f99-9c12589f86b5","Type":"ContainerStarted","Data":"ad2c2ae6b11109e62c3efb5992615a445a42bb320d783ecf8684f66e2243c2ec"}
Jul 23 22:32:36.932777 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.932419    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7" event={"ID":"5445b100-83bb-4857-8f99-9c12589f86b5","Type":"ContainerStarted","Data":"ecb104679f09c063c86d85dfc37aa8173746bf037c19f2baa0d6bbec0716aff9"}
Jul 23 22:32:36.951674 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:36.951628    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7" podStartSLOduration=1.7740302479999999 podStartE2EDuration="1.951612742s" totalImagesPullingTime="177.582494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:32:36.950413364 +0000 UTC m=+611.267942174" watchObservedRunningTime="2026-07-23 22:32:36.951612742 +0000 UTC m=+611.269141551"
Jul 23 22:32:46.111968 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:46.111910    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:46.112370 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:46.112004    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:46.124342 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:46.124318    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:32:46.987574 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:32:46.987543    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7"
Jul 23 22:35:38.339936 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.339887    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7845c898b6-544qd"]
Jul 23 22:35:38.343032 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.343014    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:38.347057 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.347027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-webhook-cert\") pod \"maas-controller-7845c898b6-544qd\" (UID: \"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523\") " pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:38.347179 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.347133    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8frc5\""
Jul 23 22:35:38.347179 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.347148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7qc\" (UniqueName: \"kubernetes.io/projected/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-kube-api-access-xr7qc\") pod \"maas-controller-7845c898b6-544qd\" (UID: \"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523\") " pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:38.347284 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.347232    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 22:35:38.347332 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.347279    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 22:35:38.352754 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.352733    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7845c898b6-544qd"]
Jul 23 22:35:38.448316 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.448279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7qc\" (UniqueName: \"kubernetes.io/projected/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-kube-api-access-xr7qc\") pod \"maas-controller-7845c898b6-544qd\" (UID: \"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523\") " pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:38.448491 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.448337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-webhook-cert\") pod \"maas-controller-7845c898b6-544qd\" (UID: \"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523\") " pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:38.450734 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.450702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-webhook-cert\") pod \"maas-controller-7845c898b6-544qd\" (UID: \"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523\") " pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:38.461757 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.461732    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7qc\" (UniqueName: \"kubernetes.io/projected/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-kube-api-access-xr7qc\") pod \"maas-controller-7845c898b6-544qd\" (UID: \"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523\") " pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:38.654993 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.654892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:38.788822 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.788795    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7845c898b6-544qd"]
Jul 23 22:35:38.790802 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:35:38.790776    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4123f6_c56c_41dd_8f22_0fa8b9c7a523.slice/crio-ba25e081e2f6adb7eabffdaa4a18dee1f383fa765b75ff0c992b75dbf583b576 WatchSource:0}: Error finding container ba25e081e2f6adb7eabffdaa4a18dee1f383fa765b75ff0c992b75dbf583b576: Status 404 returned error can't find the container with id ba25e081e2f6adb7eabffdaa4a18dee1f383fa765b75ff0c992b75dbf583b576
Jul 23 22:35:38.792554 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:38.792536    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 22:35:39.721256 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:39.721173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7845c898b6-544qd" event={"ID":"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523","Type":"ContainerStarted","Data":"ba25e081e2f6adb7eabffdaa4a18dee1f383fa765b75ff0c992b75dbf583b576"}
Jul 23 22:35:41.732530 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:41.732493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7845c898b6-544qd" event={"ID":"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523","Type":"ContainerStarted","Data":"bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845"}
Jul 23 22:35:41.732991 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:41.732618    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:41.755346 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:41.755278    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7845c898b6-544qd" podStartSLOduration=1.5268250970000001 podStartE2EDuration="3.755254446s" totalImagesPullingTime="2.228429349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:35:41.753196777 +0000 UTC m=+796.070725586" watchObservedRunningTime="2026-07-23 22:35:41.755254446 +0000 UTC m=+796.072783254"
Jul 23 22:35:52.739137 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:52.739050    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:57.438581 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.438545    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7845c898b6-544qd"]
Jul 23 22:35:57.439050 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.438753    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7845c898b6-544qd" podUID="dc4123f6-c56c-41dd-8f22-0fa8b9c7a523" containerName="manager" containerID="cri-o://bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845" gracePeriod=30
Jul 23 22:35:57.687240 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.687212    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:57.804125 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.804043    2583 generic.go:356] "Generic (PLEG): container finished" podID="dc4123f6-c56c-41dd-8f22-0fa8b9c7a523" containerID="bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845" exitCode=0
Jul 23 22:35:57.804125 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.804093    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7845c898b6-544qd" event={"ID":"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523","Type":"ContainerDied","Data":"bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845"}
Jul 23 22:35:57.804125 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.804102    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7845c898b6-544qd"
Jul 23 22:35:57.804404 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.804134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7845c898b6-544qd" event={"ID":"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523","Type":"ContainerDied","Data":"ba25e081e2f6adb7eabffdaa4a18dee1f383fa765b75ff0c992b75dbf583b576"}
Jul 23 22:35:57.804404 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.804156    2583 scope.go:117] "RemoveContainer" containerID="bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845"
Jul 23 22:35:57.817513 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.812272    2583 scope.go:117] "RemoveContainer" containerID="bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845"
Jul 23 22:35:57.817513 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:35:57.814209    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845\": container with ID starting with bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845 not found: ID does not exist" containerID="bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845"
Jul 23 22:35:57.817513 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.814245    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845"} err="failed to get container status \"bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845\": rpc error: code = NotFound desc = could not find container \"bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845\": container with ID starting with bc0000b73dc4a041b46340d0369730a7dd118a8ee907c89111c226044a45c845 not found: ID does not exist"
Jul 23 22:35:57.819720 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.819694    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-webhook-cert\") pod \"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523\" (UID: \"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523\") "
Jul 23 22:35:57.819834 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.819735    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr7qc\" (UniqueName: \"kubernetes.io/projected/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-kube-api-access-xr7qc\") pod \"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523\" (UID: \"dc4123f6-c56c-41dd-8f22-0fa8b9c7a523\") "
Jul 23 22:35:57.821709 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.821681    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dc4123f6-c56c-41dd-8f22-0fa8b9c7a523" (UID: "dc4123f6-c56c-41dd-8f22-0fa8b9c7a523"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:35:57.821709 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.821701    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-kube-api-access-xr7qc" (OuterVolumeSpecName: "kube-api-access-xr7qc") pod "dc4123f6-c56c-41dd-8f22-0fa8b9c7a523" (UID: "dc4123f6-c56c-41dd-8f22-0fa8b9c7a523"). InnerVolumeSpecName "kube-api-access-xr7qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:35:57.920970 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.920904    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-webhook-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:35:57.920970 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:57.920962    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr7qc\" (UniqueName: \"kubernetes.io/projected/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523-kube-api-access-xr7qc\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:35:58.129731 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:58.129699    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7845c898b6-544qd"]
Jul 23 22:35:58.136303 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:58.136272    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7845c898b6-544qd"]
Jul 23 22:35:58.228699 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:58.228665    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc4123f6-c56c-41dd-8f22-0fa8b9c7a523" path="/var/lib/kubelet/pods/dc4123f6-c56c-41dd-8f22-0fa8b9c7a523/volumes"
Jul 23 22:35:58.994500 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:58.994459    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7845c898b6-sphkp"]
Jul 23 22:35:58.994887 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:58.994792    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4123f6-c56c-41dd-8f22-0fa8b9c7a523" containerName="manager"
Jul 23 22:35:58.994887 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:58.994803    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4123f6-c56c-41dd-8f22-0fa8b9c7a523" containerName="manager"
Jul 23 22:35:58.994887 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:58.994854    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4123f6-c56c-41dd-8f22-0fa8b9c7a523" containerName="manager"
Jul 23 22:35:58.999187 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:58.999167    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7845c898b6-sphkp"
Jul 23 22:35:59.003528 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.003508    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8frc5\""
Jul 23 22:35:59.003528 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.003534    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 22:35:59.003686 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.003582    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 22:35:59.010514 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.010489    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7845c898b6-sphkp"]
Jul 23 22:35:59.130473 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.130431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60cdae2a-9da2-4f71-91a9-729226c067fe-webhook-cert\") pod \"maas-controller-7845c898b6-sphkp\" (UID: \"60cdae2a-9da2-4f71-91a9-729226c067fe\") " pod="opendatahub/maas-controller-7845c898b6-sphkp"
Jul 23 22:35:59.130669 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.130508    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fwl8\" (UniqueName: \"kubernetes.io/projected/60cdae2a-9da2-4f71-91a9-729226c067fe-kube-api-access-6fwl8\") pod \"maas-controller-7845c898b6-sphkp\" (UID: \"60cdae2a-9da2-4f71-91a9-729226c067fe\") " pod="opendatahub/maas-controller-7845c898b6-sphkp"
Jul 23 22:35:59.231598 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.231555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60cdae2a-9da2-4f71-91a9-729226c067fe-webhook-cert\") pod \"maas-controller-7845c898b6-sphkp\" (UID: \"60cdae2a-9da2-4f71-91a9-729226c067fe\") " pod="opendatahub/maas-controller-7845c898b6-sphkp"
Jul 23 22:35:59.231797 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.231621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fwl8\" (UniqueName: \"kubernetes.io/projected/60cdae2a-9da2-4f71-91a9-729226c067fe-kube-api-access-6fwl8\") pod \"maas-controller-7845c898b6-sphkp\" (UID: \"60cdae2a-9da2-4f71-91a9-729226c067fe\") " pod="opendatahub/maas-controller-7845c898b6-sphkp"
Jul 23 22:35:59.234076 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.234050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60cdae2a-9da2-4f71-91a9-729226c067fe-webhook-cert\") pod \"maas-controller-7845c898b6-sphkp\" (UID: \"60cdae2a-9da2-4f71-91a9-729226c067fe\") " pod="opendatahub/maas-controller-7845c898b6-sphkp"
Jul 23 22:35:59.242171 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.242151    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fwl8\" (UniqueName: \"kubernetes.io/projected/60cdae2a-9da2-4f71-91a9-729226c067fe-kube-api-access-6fwl8\") pod \"maas-controller-7845c898b6-sphkp\" (UID: \"60cdae2a-9da2-4f71-91a9-729226c067fe\") " pod="opendatahub/maas-controller-7845c898b6-sphkp"
Jul 23 22:35:59.310571 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.310532    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7845c898b6-sphkp"
Jul 23 22:35:59.453857 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.453834    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7845c898b6-sphkp"]
Jul 23 22:35:59.456259 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:35:59.456223    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cdae2a_9da2_4f71_91a9_729226c067fe.slice/crio-7c477adbaba3c36b78e4d4bba70e9eac854877443d755404bb912f1b7ac56bb4 WatchSource:0}: Error finding container 7c477adbaba3c36b78e4d4bba70e9eac854877443d755404bb912f1b7ac56bb4: Status 404 returned error can't find the container with id 7c477adbaba3c36b78e4d4bba70e9eac854877443d755404bb912f1b7ac56bb4
Jul 23 22:35:59.814905 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:35:59.814870    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7845c898b6-sphkp" event={"ID":"60cdae2a-9da2-4f71-91a9-729226c067fe","Type":"ContainerStarted","Data":"7c477adbaba3c36b78e4d4bba70e9eac854877443d755404bb912f1b7ac56bb4"}
Jul 23 22:36:00.820650 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:36:00.820614    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7845c898b6-sphkp" event={"ID":"60cdae2a-9da2-4f71-91a9-729226c067fe","Type":"ContainerStarted","Data":"664b1cc66276339512dbe1d29bca2270b89c2b535b537a411ad5a76bd1456238"}
Jul 23 22:36:00.821069 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:36:00.820717    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7845c898b6-sphkp"
Jul 23 22:36:00.842543 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:36:00.842490    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7845c898b6-sphkp" podStartSLOduration=2.458694519 podStartE2EDuration="2.842473871s" totalImagesPullingTime="383.779352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:36:00.840689587 +0000 UTC m=+815.158218399" watchObservedRunningTime="2026-07-23 22:36:00.842473871 +0000 UTC m=+815.160002720"
Jul 23 22:36:11.827170 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:36:11.827138    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7845c898b6-sphkp"
Jul 23 22:37:26.177606 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:37:26.177578    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:37:26.181219 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:37:26.181187    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:42:26.205662 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:42:26.205630    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:42:26.206544 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:42:26.206524    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:45:00.144504 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:00.144425    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr"]
Jul 23 22:45:00.147820 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:00.147793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr"
Jul 23 22:45:00.150406 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:00.150380    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6cq4\""
Jul 23 22:45:00.156366 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:00.156342    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr"]
Jul 23 22:45:00.158702 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:00.158683    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr"
Jul 23 22:45:00.288892 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:00.288870    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr"]
Jul 23 22:45:00.291451 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:45:00.291421    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f6172e_8bf6_44cd_b94b_0153a2552a97.slice/crio-dd32a5994bfc4e540e97ade47880da2c93fdea7c59a63c653b37d506812605a6 WatchSource:0}: Error finding container dd32a5994bfc4e540e97ade47880da2c93fdea7c59a63c653b37d506812605a6: Status 404 returned error can't find the container with id dd32a5994bfc4e540e97ade47880da2c93fdea7c59a63c653b37d506812605a6
Jul 23 22:45:00.293414 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:00.293391    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 22:45:01.182249 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:01.182200    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr" event={"ID":"a0f6172e-8bf6-44cd-b94b-0153a2552a97","Type":"ContainerStarted","Data":"dd32a5994bfc4e540e97ade47880da2c93fdea7c59a63c653b37d506812605a6"}
Jul 23 22:45:03.193642 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:03.193605    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr" event={"ID":"a0f6172e-8bf6-44cd-b94b-0153a2552a97","Type":"ContainerStarted","Data":"a5c3aae4a3c4bd09e7d960500aeb67a00607e4e96eff728106bdc0984d9709b0"}
Jul 23 22:45:03.210261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:03.210212    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr" podStartSLOduration=1.143726736 podStartE2EDuration="3.210196405s" totalImagesPullingTime="2.066469669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:45:03.208670589 +0000 UTC m=+1357.526199398" watchObservedRunningTime="2026-07-23 22:45:03.210196405 +0000 UTC m=+1357.527725213"
Jul 23 22:45:04.199457 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:04.199419    2583 generic.go:356] "Generic (PLEG): container finished" podID="a0f6172e-8bf6-44cd-b94b-0153a2552a97" containerID="a5c3aae4a3c4bd09e7d960500aeb67a00607e4e96eff728106bdc0984d9709b0" exitCode=0
Jul 23 22:45:04.199862 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:04.199463    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr" event={"ID":"a0f6172e-8bf6-44cd-b94b-0153a2552a97","Type":"ContainerDied","Data":"a5c3aae4a3c4bd09e7d960500aeb67a00607e4e96eff728106bdc0984d9709b0"}
Jul 23 22:45:05.332175 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:05.332151    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr"
Jul 23 22:45:06.212082 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:06.212049    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr"
Jul 23 22:45:06.212082 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:06.212068    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747445-76whr" event={"ID":"a0f6172e-8bf6-44cd-b94b-0153a2552a97","Type":"ContainerDied","Data":"dd32a5994bfc4e540e97ade47880da2c93fdea7c59a63c653b37d506812605a6"}
Jul 23 22:45:06.212289 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:45:06.212136    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd32a5994bfc4e540e97ade47880da2c93fdea7c59a63c653b37d506812605a6"
Jul 23 22:47:26.233532 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:47:26.233503    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:47:26.234449 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:47:26.234430    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:48:58.715933 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.715879    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"]
Jul 23 22:48:58.716460 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.716439    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f6172e-8bf6-44cd-b94b-0153a2552a97" containerName="cleanup"
Jul 23 22:48:58.716521 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.716465    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f6172e-8bf6-44cd-b94b-0153a2552a97" containerName="cleanup"
Jul 23 22:48:58.716576 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.716565    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f6172e-8bf6-44cd-b94b-0153a2552a97" containerName="cleanup"
Jul 23 22:48:58.719903 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.719882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:48:58.723123 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.722915    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 22:48:58.723123 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.722995    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f205eed2-serving-cert\""
Jul 23 22:48:58.723123 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.723004    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 22:48:58.723123 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.723018    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vw8vx\""
Jul 23 22:48:58.729812 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.729745    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"]
Jul 23 22:48:58.876427 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.876386    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6362d3cd-1481-4973-a45e-5dc3933388bd-openshift-service-ca\") pod \"maas-api-e2e-ait-f205eed2-884d56448-tjqk4\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:48:58.876633 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.876433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6362d3cd-1481-4973-a45e-5dc3933388bd-maas-api-tls\") pod \"maas-api-e2e-ait-f205eed2-884d56448-tjqk4\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:48:58.876633 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.876514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpvgg\" (UniqueName: \"kubernetes.io/projected/6362d3cd-1481-4973-a45e-5dc3933388bd-kube-api-access-rpvgg\") pod \"maas-api-e2e-ait-f205eed2-884d56448-tjqk4\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:48:58.977450 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.977353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpvgg\" (UniqueName: \"kubernetes.io/projected/6362d3cd-1481-4973-a45e-5dc3933388bd-kube-api-access-rpvgg\") pod \"maas-api-e2e-ait-f205eed2-884d56448-tjqk4\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:48:58.977625 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.977465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6362d3cd-1481-4973-a45e-5dc3933388bd-openshift-service-ca\") pod \"maas-api-e2e-ait-f205eed2-884d56448-tjqk4\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:48:58.977625 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.977505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6362d3cd-1481-4973-a45e-5dc3933388bd-maas-api-tls\") pod \"maas-api-e2e-ait-f205eed2-884d56448-tjqk4\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:48:58.978215 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.978188    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6362d3cd-1481-4973-a45e-5dc3933388bd-openshift-service-ca\") pod \"maas-api-e2e-ait-f205eed2-884d56448-tjqk4\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:48:58.980122 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.980093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6362d3cd-1481-4973-a45e-5dc3933388bd-maas-api-tls\") pod \"maas-api-e2e-ait-f205eed2-884d56448-tjqk4\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:48:58.988619 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:58.988598    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpvgg\" (UniqueName: \"kubernetes.io/projected/6362d3cd-1481-4973-a45e-5dc3933388bd-kube-api-access-rpvgg\") pod \"maas-api-e2e-ait-f205eed2-884d56448-tjqk4\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:48:59.032163 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.032127    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:48:59.164389 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.164362    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"]
Jul 23 22:48:59.166653 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:48:59.166623    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6362d3cd_1481_4973_a45e_5dc3933388bd.slice/crio-cfe9f632cc6c3e451dfd0522c9e13e8c8aeb10d147185658288958af0f02e2ff WatchSource:0}: Error finding container cfe9f632cc6c3e451dfd0522c9e13e8c8aeb10d147185658288958af0f02e2ff: Status 404 returned error can't find the container with id cfe9f632cc6c3e451dfd0522c9e13e8c8aeb10d147185658288958af0f02e2ff
Jul 23 22:48:59.256779 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.256687    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4" event={"ID":"6362d3cd-1481-4973-a45e-5dc3933388bd","Type":"ContainerStarted","Data":"cfe9f632cc6c3e451dfd0522c9e13e8c8aeb10d147185658288958af0f02e2ff"}
Jul 23 22:48:59.472118 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.472081    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"]
Jul 23 22:48:59.476668 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.476646    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:48:59.479290 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.479263    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f205eed2-dockercfg-7mvbs\""
Jul 23 22:48:59.479431 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.479292    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f205eed2\""
Jul 23 22:48:59.486191 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.486166    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"]
Jul 23 22:48:59.508055 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.507977    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"]
Jul 23 22:48:59.511402 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.511383    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:48:59.522185 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.522151    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"]
Jul 23 22:48:59.582359 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.582326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ckq\" (UniqueName: \"kubernetes.io/projected/0caff45f-6240-404f-b7fe-018b53cc6087-kube-api-access-87ckq\") pod \"payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr\" (UID: \"0caff45f-6240-404f-b7fe-018b53cc6087\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:48:59.582528 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.582377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0caff45f-6240-404f-b7fe-018b53cc6087-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr\" (UID: \"0caff45f-6240-404f-b7fe-018b53cc6087\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:48:59.582528 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.582438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2kq8\" (UniqueName: \"kubernetes.io/projected/c4823716-1113-4ea0-8439-325f79cf52c5-kube-api-access-p2kq8\") pod \"payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc\" (UID: \"c4823716-1113-4ea0-8439-325f79cf52c5\") " pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:48:59.582601 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.582557    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4823716-1113-4ea0-8439-325f79cf52c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc\" (UID: \"c4823716-1113-4ea0-8439-325f79cf52c5\") " pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:48:59.684011 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.683978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4823716-1113-4ea0-8439-325f79cf52c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc\" (UID: \"c4823716-1113-4ea0-8439-325f79cf52c5\") " pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:48:59.684197 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.684032    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ckq\" (UniqueName: \"kubernetes.io/projected/0caff45f-6240-404f-b7fe-018b53cc6087-kube-api-access-87ckq\") pod \"payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr\" (UID: \"0caff45f-6240-404f-b7fe-018b53cc6087\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:48:59.684197 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.684069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0caff45f-6240-404f-b7fe-018b53cc6087-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr\" (UID: \"0caff45f-6240-404f-b7fe-018b53cc6087\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:48:59.684197 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.684089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2kq8\" (UniqueName: \"kubernetes.io/projected/c4823716-1113-4ea0-8439-325f79cf52c5-kube-api-access-p2kq8\") pod \"payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc\" (UID: \"c4823716-1113-4ea0-8439-325f79cf52c5\") " pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:48:59.684818 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.684794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4823716-1113-4ea0-8439-325f79cf52c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc\" (UID: \"c4823716-1113-4ea0-8439-325f79cf52c5\") " pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:48:59.684882 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.684802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0caff45f-6240-404f-b7fe-018b53cc6087-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr\" (UID: \"0caff45f-6240-404f-b7fe-018b53cc6087\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:48:59.692249 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.692215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ckq\" (UniqueName: \"kubernetes.io/projected/0caff45f-6240-404f-b7fe-018b53cc6087-kube-api-access-87ckq\") pod \"payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr\" (UID: \"0caff45f-6240-404f-b7fe-018b53cc6087\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:48:59.692353 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.692215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2kq8\" (UniqueName: \"kubernetes.io/projected/c4823716-1113-4ea0-8439-325f79cf52c5-kube-api-access-p2kq8\") pod \"payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc\" (UID: \"c4823716-1113-4ea0-8439-325f79cf52c5\") " pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:48:59.788287 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.788191    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:48:59.822529 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.822490    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:48:59.960674 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.960640    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"]
Jul 23 22:48:59.964461 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:48:59.964417    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0caff45f_6240_404f_b7fe_018b53cc6087.slice/crio-11cd14461bead7e22df47cdaa6a181f3135f62a762314a86cd86232a37b07a17 WatchSource:0}: Error finding container 11cd14461bead7e22df47cdaa6a181f3135f62a762314a86cd86232a37b07a17: Status 404 returned error can't find the container with id 11cd14461bead7e22df47cdaa6a181f3135f62a762314a86cd86232a37b07a17
Jul 23 22:48:59.992095 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:48:59.991886    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"]
Jul 23 22:48:59.995347 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:48:59.995314    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4823716_1113_4ea0_8439_325f79cf52c5.slice/crio-033c73cadfb2384a5c66385e35c843a4300410e1d54d65c57f7a0727df698ff2 WatchSource:0}: Error finding container 033c73cadfb2384a5c66385e35c843a4300410e1d54d65c57f7a0727df698ff2: Status 404 returned error can't find the container with id 033c73cadfb2384a5c66385e35c843a4300410e1d54d65c57f7a0727df698ff2
Jul 23 22:49:00.263498 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:00.263458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc" event={"ID":"c4823716-1113-4ea0-8439-325f79cf52c5","Type":"ContainerStarted","Data":"033c73cadfb2384a5c66385e35c843a4300410e1d54d65c57f7a0727df698ff2"}
Jul 23 22:49:00.264569 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:00.264545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr" event={"ID":"0caff45f-6240-404f-b7fe-018b53cc6087","Type":"ContainerStarted","Data":"11cd14461bead7e22df47cdaa6a181f3135f62a762314a86cd86232a37b07a17"}
Jul 23 22:49:01.281527 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:01.281102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4" event={"ID":"6362d3cd-1481-4973-a45e-5dc3933388bd","Type":"ContainerStarted","Data":"881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63"}
Jul 23 22:49:01.282053 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:01.281720    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:49:01.302096 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:01.302007    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4" podStartSLOduration=1.721563006 podStartE2EDuration="3.301982734s" totalImagesPullingTime="1.580419728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:49:01.301163234 +0000 UTC m=+1595.618692044" watchObservedRunningTime="2026-07-23 22:49:01.301982734 +0000 UTC m=+1595.619511546"
Jul 23 22:49:03.292991 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:03.292950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr" event={"ID":"0caff45f-6240-404f-b7fe-018b53cc6087","Type":"ContainerStarted","Data":"c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210"}
Jul 23 22:49:03.293450 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:03.293017    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:49:03.294526 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:03.294488    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc" event={"ID":"c4823716-1113-4ea0-8439-325f79cf52c5","Type":"ContainerStarted","Data":"188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7"}
Jul 23 22:49:03.294683 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:03.294664    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:49:03.310391 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:03.310342    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr" podStartSLOduration=1.495938653 podStartE2EDuration="4.310323499s" totalImagesPullingTime="2.814384846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:49:03.309492287 +0000 UTC m=+1597.627021097" watchObservedRunningTime="2026-07-23 22:49:03.310323499 +0000 UTC m=+1597.627852307"
Jul 23 22:49:03.326136 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:03.326083    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc" podStartSLOduration=1.547625316 podStartE2EDuration="4.326067368s" totalImagesPullingTime="2.778442052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:49:03.324253328 +0000 UTC m=+1597.641782137" watchObservedRunningTime="2026-07-23 22:49:03.326067368 +0000 UTC m=+1597.643596176"
Jul 23 22:49:08.301078 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:08.301042    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:49:14.301329 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:14.301299    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:49:14.301733 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:14.301368    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:49:24.044621 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:24.044536    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"]
Jul 23 22:49:24.049145 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:24.049122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"
Jul 23 22:49:24.052403 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:24.052376    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6cq4\""
Jul 23 22:49:24.055153 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:24.055119    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"]
Jul 23 22:49:24.103595 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:24.103562    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f585ffb4-7bb5-49da-83bc-5838e991f469-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b\" (UID: \"f585ffb4-7bb5-49da-83bc-5838e991f469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"
Jul 23 22:49:24.204341 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:24.204297    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f585ffb4-7bb5-49da-83bc-5838e991f469-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b\" (UID: \"f585ffb4-7bb5-49da-83bc-5838e991f469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"
Jul 23 22:49:24.205020 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:24.205002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f585ffb4-7bb5-49da-83bc-5838e991f469-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b\" (UID: \"f585ffb4-7bb5-49da-83bc-5838e991f469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"
Jul 23 22:49:24.362261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:24.362226    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"
Jul 23 22:49:24.500092 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:24.499944    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"]
Jul 23 22:49:24.503099 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:49:24.503057    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf585ffb4_7bb5_49da_83bc_5838e991f469.slice/crio-5177c392495955dfd7d168db1e8859f06042473409ae7c6daa99379c92a1d8cb WatchSource:0}: Error finding container 5177c392495955dfd7d168db1e8859f06042473409ae7c6daa99379c92a1d8cb: Status 404 returned error can't find the container with id 5177c392495955dfd7d168db1e8859f06042473409ae7c6daa99379c92a1d8cb
Jul 23 22:49:25.399141 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:25.399105    2583 generic.go:356] "Generic (PLEG): container finished" podID="f585ffb4-7bb5-49da-83bc-5838e991f469" containerID="86a6be115efe6b661dc4d8972cf4a7358dd83d121fbe9274dbaf4d3bc703f734" exitCode=0
Jul 23 22:49:25.399530 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:25.399147    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b" event={"ID":"f585ffb4-7bb5-49da-83bc-5838e991f469","Type":"ContainerDied","Data":"86a6be115efe6b661dc4d8972cf4a7358dd83d121fbe9274dbaf4d3bc703f734"}
Jul 23 22:49:25.399530 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:25.399217    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b" event={"ID":"f585ffb4-7bb5-49da-83bc-5838e991f469","Type":"ContainerStarted","Data":"5177c392495955dfd7d168db1e8859f06042473409ae7c6daa99379c92a1d8cb"}
Jul 23 22:49:26.535815 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:26.535787    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"
Jul 23 22:49:26.632440 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:26.632397    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f585ffb4-7bb5-49da-83bc-5838e991f469-maas-api-service-ca\") pod \"f585ffb4-7bb5-49da-83bc-5838e991f469\" (UID: \"f585ffb4-7bb5-49da-83bc-5838e991f469\") "
Jul 23 22:49:26.632773 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:26.632748    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f585ffb4-7bb5-49da-83bc-5838e991f469-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f585ffb4-7bb5-49da-83bc-5838e991f469" (UID: "f585ffb4-7bb5-49da-83bc-5838e991f469"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:49:26.733271 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:26.733181    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f585ffb4-7bb5-49da-83bc-5838e991f469-maas-api-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:49:27.410268 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:27.410227    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"
Jul 23 22:49:27.410420 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:27.410238    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b" event={"ID":"f585ffb4-7bb5-49da-83bc-5838e991f469","Type":"ContainerDied","Data":"5177c392495955dfd7d168db1e8859f06042473409ae7c6daa99379c92a1d8cb"}
Jul 23 22:49:27.410420 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:27.410358    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5177c392495955dfd7d168db1e8859f06042473409ae7c6daa99379c92a1d8cb"
Jul 23 22:49:34.465180 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.465142    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"]
Jul 23 22:49:34.465739 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.465380    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4" podUID="6362d3cd-1481-4973-a45e-5dc3933388bd" containerName="maas-api" containerID="cri-o://881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63" gracePeriod=30
Jul 23 22:49:34.517080 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.517039    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"]
Jul 23 22:49:34.517379 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.517321    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc" podUID="c4823716-1113-4ea0-8439-325f79cf52c5" containerName="payload-processing" containerID="cri-o://188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7" gracePeriod=30
Jul 23 22:49:34.524744 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.524629    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"]
Jul 23 22:49:34.525027 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.524945    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr" podUID="0caff45f-6240-404f-b7fe-018b53cc6087" containerName="payload-pre-processing" containerID="cri-o://c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210" gracePeriod=30
Jul 23 22:49:34.796595 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.796561    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:49:34.817062 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.817013    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:49:34.826354 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.826331    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:49:34.905144 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.905105    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87ckq\" (UniqueName: \"kubernetes.io/projected/0caff45f-6240-404f-b7fe-018b53cc6087-kube-api-access-87ckq\") pod \"0caff45f-6240-404f-b7fe-018b53cc6087\" (UID: \"0caff45f-6240-404f-b7fe-018b53cc6087\") "
Jul 23 22:49:34.905357 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.905154    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2kq8\" (UniqueName: \"kubernetes.io/projected/c4823716-1113-4ea0-8439-325f79cf52c5-kube-api-access-p2kq8\") pod \"c4823716-1113-4ea0-8439-325f79cf52c5\" (UID: \"c4823716-1113-4ea0-8439-325f79cf52c5\") "
Jul 23 22:49:34.905357 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.905204    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0caff45f-6240-404f-b7fe-018b53cc6087-plugins-config-volume\") pod \"0caff45f-6240-404f-b7fe-018b53cc6087\" (UID: \"0caff45f-6240-404f-b7fe-018b53cc6087\") "
Jul 23 22:49:34.905485 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.905354    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6362d3cd-1481-4973-a45e-5dc3933388bd-maas-api-tls\") pod \"6362d3cd-1481-4973-a45e-5dc3933388bd\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") "
Jul 23 22:49:34.905485 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.905399    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6362d3cd-1481-4973-a45e-5dc3933388bd-openshift-service-ca\") pod \"6362d3cd-1481-4973-a45e-5dc3933388bd\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") "
Jul 23 22:49:34.905485 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.905467    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpvgg\" (UniqueName: \"kubernetes.io/projected/6362d3cd-1481-4973-a45e-5dc3933388bd-kube-api-access-rpvgg\") pod \"6362d3cd-1481-4973-a45e-5dc3933388bd\" (UID: \"6362d3cd-1481-4973-a45e-5dc3933388bd\") "
Jul 23 22:49:34.905641 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.905506    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4823716-1113-4ea0-8439-325f79cf52c5-plugins-config-volume\") pod \"c4823716-1113-4ea0-8439-325f79cf52c5\" (UID: \"c4823716-1113-4ea0-8439-325f79cf52c5\") "
Jul 23 22:49:34.905641 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.905593    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0caff45f-6240-404f-b7fe-018b53cc6087-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0caff45f-6240-404f-b7fe-018b53cc6087" (UID: "0caff45f-6240-404f-b7fe-018b53cc6087"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:49:34.905939 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.905853    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6362d3cd-1481-4973-a45e-5dc3933388bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6362d3cd-1481-4973-a45e-5dc3933388bd" (UID: "6362d3cd-1481-4973-a45e-5dc3933388bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:49:34.906041 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.905879    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0caff45f-6240-404f-b7fe-018b53cc6087-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:49:34.906292 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.906265    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4823716-1113-4ea0-8439-325f79cf52c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4823716-1113-4ea0-8439-325f79cf52c5" (UID: "c4823716-1113-4ea0-8439-325f79cf52c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:49:34.907692 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.907671    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6362d3cd-1481-4973-a45e-5dc3933388bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6362d3cd-1481-4973-a45e-5dc3933388bd" (UID: "6362d3cd-1481-4973-a45e-5dc3933388bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:49:34.907779 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.907760    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6362d3cd-1481-4973-a45e-5dc3933388bd-kube-api-access-rpvgg" (OuterVolumeSpecName: "kube-api-access-rpvgg") pod "6362d3cd-1481-4973-a45e-5dc3933388bd" (UID: "6362d3cd-1481-4973-a45e-5dc3933388bd"). InnerVolumeSpecName "kube-api-access-rpvgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:49:34.907963 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.907943    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4823716-1113-4ea0-8439-325f79cf52c5-kube-api-access-p2kq8" (OuterVolumeSpecName: "kube-api-access-p2kq8") pod "c4823716-1113-4ea0-8439-325f79cf52c5" (UID: "c4823716-1113-4ea0-8439-325f79cf52c5"). InnerVolumeSpecName "kube-api-access-p2kq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:49:34.908010 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:34.907969    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0caff45f-6240-404f-b7fe-018b53cc6087-kube-api-access-87ckq" (OuterVolumeSpecName: "kube-api-access-87ckq") pod "0caff45f-6240-404f-b7fe-018b53cc6087" (UID: "0caff45f-6240-404f-b7fe-018b53cc6087"). InnerVolumeSpecName "kube-api-access-87ckq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:49:35.006882 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.006801    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87ckq\" (UniqueName: \"kubernetes.io/projected/0caff45f-6240-404f-b7fe-018b53cc6087-kube-api-access-87ckq\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:49:35.006882 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.006834    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2kq8\" (UniqueName: \"kubernetes.io/projected/c4823716-1113-4ea0-8439-325f79cf52c5-kube-api-access-p2kq8\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:49:35.006882 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.006848    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6362d3cd-1481-4973-a45e-5dc3933388bd-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:49:35.006882 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.006856    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6362d3cd-1481-4973-a45e-5dc3933388bd-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:49:35.006882 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.006872    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpvgg\" (UniqueName: \"kubernetes.io/projected/6362d3cd-1481-4973-a45e-5dc3933388bd-kube-api-access-rpvgg\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:49:35.006882 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.006881    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4823716-1113-4ea0-8439-325f79cf52c5-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:49:35.450884 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.450821    2583 generic.go:356] "Generic (PLEG): container finished" podID="c4823716-1113-4ea0-8439-325f79cf52c5" containerID="188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7" exitCode=0
Jul 23 22:49:35.451112 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.451017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc" event={"ID":"c4823716-1113-4ea0-8439-325f79cf52c5","Type":"ContainerDied","Data":"188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7"}
Jul 23 22:49:35.451178 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.451113    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc" event={"ID":"c4823716-1113-4ea0-8439-325f79cf52c5","Type":"ContainerDied","Data":"033c73cadfb2384a5c66385e35c843a4300410e1d54d65c57f7a0727df698ff2"}
Jul 23 22:49:35.451178 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.451150    2583 scope.go:117] "RemoveContainer" containerID="188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7"
Jul 23 22:49:35.453396 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.451340    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"
Jul 23 22:49:35.455415 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.455384    2583 generic.go:356] "Generic (PLEG): container finished" podID="0caff45f-6240-404f-b7fe-018b53cc6087" containerID="c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210" exitCode=0
Jul 23 22:49:35.455532 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.455446    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"
Jul 23 22:49:35.455532 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.455460    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr" event={"ID":"0caff45f-6240-404f-b7fe-018b53cc6087","Type":"ContainerDied","Data":"c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210"}
Jul 23 22:49:35.455532 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.455526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr" event={"ID":"0caff45f-6240-404f-b7fe-018b53cc6087","Type":"ContainerDied","Data":"11cd14461bead7e22df47cdaa6a181f3135f62a762314a86cd86232a37b07a17"}
Jul 23 22:49:35.457043 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.457016    2583 generic.go:356] "Generic (PLEG): container finished" podID="6362d3cd-1481-4973-a45e-5dc3933388bd" containerID="881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63" exitCode=0
Jul 23 22:49:35.457155 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.457048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4" event={"ID":"6362d3cd-1481-4973-a45e-5dc3933388bd","Type":"ContainerDied","Data":"881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63"}
Jul 23 22:49:35.457155 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.457091    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"
Jul 23 22:49:35.457155 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.457108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4" event={"ID":"6362d3cd-1481-4973-a45e-5dc3933388bd","Type":"ContainerDied","Data":"cfe9f632cc6c3e451dfd0522c9e13e8c8aeb10d147185658288958af0f02e2ff"}
Jul 23 22:49:35.464393 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.464369    2583 scope.go:117] "RemoveContainer" containerID="188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7"
Jul 23 22:49:35.464718 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:49:35.464696    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7\": container with ID starting with 188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7 not found: ID does not exist" containerID="188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7"
Jul 23 22:49:35.464813 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.464724    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7"} err="failed to get container status \"188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7\": rpc error: code = NotFound desc = could not find container \"188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7\": container with ID starting with 188d76b443ebb7e90977806013d094b2903c65907ae7996eb6cde2521e25ebf7 not found: ID does not exist"
Jul 23 22:49:35.464813 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.464743    2583 scope.go:117] "RemoveContainer" containerID="c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210"
Jul 23 22:49:35.473803 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.473658    2583 scope.go:117] "RemoveContainer" containerID="c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210"
Jul 23 22:49:35.474103 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:49:35.473976    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210\": container with ID starting with c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210 not found: ID does not exist" containerID="c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210"
Jul 23 22:49:35.474103 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.474012    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210"} err="failed to get container status \"c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210\": rpc error: code = NotFound desc = could not find container \"c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210\": container with ID starting with c2a5fd60f78c14d0c26121412f15126e9cfbfb4f1a416691acc9220ff0479210 not found: ID does not exist"
Jul 23 22:49:35.474103 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.474032    2583 scope.go:117] "RemoveContainer" containerID="881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63"
Jul 23 22:49:35.478378 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.478351    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"]
Jul 23 22:49:35.481436 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.481412    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f205eed2-648c6ddc58-wj7vc"]
Jul 23 22:49:35.492776 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.492738    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"]
Jul 23 22:49:35.494860 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.494834    2583 scope.go:117] "RemoveContainer" containerID="881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63"
Jul 23 22:49:35.495378 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:49:35.495356    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63\": container with ID starting with 881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63 not found: ID does not exist" containerID="881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63"
Jul 23 22:49:35.495456 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.495386    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63"} err="failed to get container status \"881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63\": rpc error: code = NotFound desc = could not find container \"881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63\": container with ID starting with 881a7d904b6e6c72849d950ca80eeeaaece93022912775ee9b160e4ae65e1b63 not found: ID does not exist"
Jul 23 22:49:35.496597 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.496577    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f205eed2-884d56448-tjqk4"]
Jul 23 22:49:35.506550 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.506527    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"]
Jul 23 22:49:35.510249 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:35.510224    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f205eed2-77b497b4d4-j5tnr"]
Jul 23 22:49:36.229256 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:36.229217    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0caff45f-6240-404f-b7fe-018b53cc6087" path="/var/lib/kubelet/pods/0caff45f-6240-404f-b7fe-018b53cc6087/volumes"
Jul 23 22:49:36.229616 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:36.229603    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6362d3cd-1481-4973-a45e-5dc3933388bd" path="/var/lib/kubelet/pods/6362d3cd-1481-4973-a45e-5dc3933388bd/volumes"
Jul 23 22:49:36.229936 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:36.229913    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4823716-1113-4ea0-8439-325f79cf52c5" path="/var/lib/kubelet/pods/c4823716-1113-4ea0-8439-325f79cf52c5/volumes"
Jul 23 22:49:39.127715 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:39.127679    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"]
Jul 23 22:49:39.130341 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:39.130321    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f205eed2-6bb9b"]
Jul 23 22:49:40.230261 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:40.230222    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f585ffb4-7bb5-49da-83bc-5838e991f469" path="/var/lib/kubelet/pods/f585ffb4-7bb5-49da-83bc-5838e991f469/volumes"
Jul 23 22:49:42.822286 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822250    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"]
Jul 23 22:49:42.822727 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822711    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f585ffb4-7bb5-49da-83bc-5838e991f469" containerName="revoke-keys"
Jul 23 22:49:42.822727 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822729    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f585ffb4-7bb5-49da-83bc-5838e991f469" containerName="revoke-keys"
Jul 23 22:49:42.822839 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822747    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0caff45f-6240-404f-b7fe-018b53cc6087" containerName="payload-pre-processing"
Jul 23 22:49:42.822839 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822756    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0caff45f-6240-404f-b7fe-018b53cc6087" containerName="payload-pre-processing"
Jul 23 22:49:42.822839 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822782    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4823716-1113-4ea0-8439-325f79cf52c5" containerName="payload-processing"
Jul 23 22:49:42.822839 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822792    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4823716-1113-4ea0-8439-325f79cf52c5" containerName="payload-processing"
Jul 23 22:49:42.822839 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822805    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6362d3cd-1481-4973-a45e-5dc3933388bd" containerName="maas-api"
Jul 23 22:49:42.822839 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822814    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6362d3cd-1481-4973-a45e-5dc3933388bd" containerName="maas-api"
Jul 23 22:49:42.823050 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822903    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0caff45f-6240-404f-b7fe-018b53cc6087" containerName="payload-pre-processing"
Jul 23 22:49:42.823050 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822915    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4823716-1113-4ea0-8439-325f79cf52c5" containerName="payload-processing"
Jul 23 22:49:42.823050 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822952    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6362d3cd-1481-4973-a45e-5dc3933388bd" containerName="maas-api"
Jul 23 22:49:42.823050 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.822967    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f585ffb4-7bb5-49da-83bc-5838e991f469" containerName="revoke-keys"
Jul 23 22:49:42.826388 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.826366    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:42.828847 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.828824    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c9c6cb93-serving-cert\""
Jul 23 22:49:42.829000 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.828889    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 22:49:42.829000 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.828965    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 22:49:42.830052 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.830031    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vw8vx\""
Jul 23 22:49:42.834665 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.834597    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"]
Jul 23 22:49:42.974784 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.974743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2xp4\" (UniqueName: \"kubernetes.io/projected/54b3f355-86af-4839-b5e4-5fb9f270ab92-kube-api-access-t2xp4\") pod \"maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:42.975004 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.974820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b3f355-86af-4839-b5e4-5fb9f270ab92-maas-api-tls\") pod \"maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:42.975004 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:42.974874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b3f355-86af-4839-b5e4-5fb9f270ab92-openshift-service-ca\") pod \"maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:43.076030 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.075909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2xp4\" (UniqueName: \"kubernetes.io/projected/54b3f355-86af-4839-b5e4-5fb9f270ab92-kube-api-access-t2xp4\") pod \"maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:43.076030 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.076013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b3f355-86af-4839-b5e4-5fb9f270ab92-maas-api-tls\") pod \"maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:43.076227 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.076047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b3f355-86af-4839-b5e4-5fb9f270ab92-openshift-service-ca\") pod \"maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:43.076227 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:49:43.076160    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-serving-cert: secret "maas-api-e2e-migrate-c9c6cb93-serving-cert" not found
Jul 23 22:49:43.076312 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:49:43.076246    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b3f355-86af-4839-b5e4-5fb9f270ab92-maas-api-tls podName:54b3f355-86af-4839-b5e4-5fb9f270ab92 nodeName:}" failed. No retries permitted until 2026-07-23 22:49:43.576224433 +0000 UTC m=+1637.893753234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54b3f355-86af-4839-b5e4-5fb9f270ab92-maas-api-tls") pod "maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn" (UID: "54b3f355-86af-4839-b5e4-5fb9f270ab92") : secret "maas-api-e2e-migrate-c9c6cb93-serving-cert" not found
Jul 23 22:49:43.076767 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.076745    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b3f355-86af-4839-b5e4-5fb9f270ab92-openshift-service-ca\") pod \"maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:43.086453 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.086416    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2xp4\" (UniqueName: \"kubernetes.io/projected/54b3f355-86af-4839-b5e4-5fb9f270ab92-kube-api-access-t2xp4\") pod \"maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:43.536052 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.535941    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"]
Jul 23 22:49:43.539495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.539470    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:49:43.541936 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.541900    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c9c6cb93-dockercfg-5l4vh\""
Jul 23 22:49:43.542073 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.541975    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c9c6cb93\""
Jul 23 22:49:43.549457 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.549431    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"]
Jul 23 22:49:43.566433 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.566406    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"]
Jul 23 22:49:43.569724 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.569702    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:49:43.581825 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.581799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b3f355-86af-4839-b5e4-5fb9f270ab92-maas-api-tls\") pod \"maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:43.582952 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.582907    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"]
Jul 23 22:49:43.584429 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.584400    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b3f355-86af-4839-b5e4-5fb9f270ab92-maas-api-tls\") pod \"maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:43.682383 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.682335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10c6c92-2a89-4235-982d-bc4c6035a695-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m\" (UID: \"c10c6c92-2a89-4235-982d-bc4c6035a695\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:49:43.682575 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.682407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg\" (UID: \"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:49:43.682575 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.682522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjppp\" (UniqueName: \"kubernetes.io/projected/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-kube-api-access-jjppp\") pod \"payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg\" (UID: \"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:49:43.682575 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.682567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8wzv\" (UniqueName: \"kubernetes.io/projected/c10c6c92-2a89-4235-982d-bc4c6035a695-kube-api-access-q8wzv\") pod \"payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m\" (UID: \"c10c6c92-2a89-4235-982d-bc4c6035a695\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:49:43.738717 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.738678    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:43.784008 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.783967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg\" (UID: \"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:49:43.784221 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.784051    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjppp\" (UniqueName: \"kubernetes.io/projected/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-kube-api-access-jjppp\") pod \"payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg\" (UID: \"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:49:43.784221 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.784080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8wzv\" (UniqueName: \"kubernetes.io/projected/c10c6c92-2a89-4235-982d-bc4c6035a695-kube-api-access-q8wzv\") pod \"payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m\" (UID: \"c10c6c92-2a89-4235-982d-bc4c6035a695\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:49:43.784221 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.784165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10c6c92-2a89-4235-982d-bc4c6035a695-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m\" (UID: \"c10c6c92-2a89-4235-982d-bc4c6035a695\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:49:43.784839 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.784810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg\" (UID: \"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:49:43.784969 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.784895    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10c6c92-2a89-4235-982d-bc4c6035a695-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m\" (UID: \"c10c6c92-2a89-4235-982d-bc4c6035a695\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:49:43.793454 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.793399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjppp\" (UniqueName: \"kubernetes.io/projected/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-kube-api-access-jjppp\") pod \"payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg\" (UID: \"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:49:43.802219 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.802160    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8wzv\" (UniqueName: \"kubernetes.io/projected/c10c6c92-2a89-4235-982d-bc4c6035a695-kube-api-access-q8wzv\") pod \"payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m\" (UID: \"c10c6c92-2a89-4235-982d-bc4c6035a695\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:49:43.851306 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.851273    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:49:43.879890 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.879821    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:49:43.904179 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:43.904147    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"]
Jul 23 22:49:43.906272 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:49:43.906207    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b3f355_86af_4839_b5e4_5fb9f270ab92.slice/crio-e38b154955376a07368f1d9c466dd6260e15278a431446acb7fc55672218753a WatchSource:0}: Error finding container e38b154955376a07368f1d9c466dd6260e15278a431446acb7fc55672218753a: Status 404 returned error can't find the container with id e38b154955376a07368f1d9c466dd6260e15278a431446acb7fc55672218753a
Jul 23 22:49:44.005040 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.005008    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"]
Jul 23 22:49:44.007166 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:49:44.007140    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10c6c92_2a89_4235_982d_bc4c6035a695.slice/crio-3dd68cb6b17a63fdc239353fda910a47e8c6bd93672b3c5e93de7deb81d0ac3d WatchSource:0}: Error finding container 3dd68cb6b17a63fdc239353fda910a47e8c6bd93672b3c5e93de7deb81d0ac3d: Status 404 returned error can't find the container with id 3dd68cb6b17a63fdc239353fda910a47e8c6bd93672b3c5e93de7deb81d0ac3d
Jul 23 22:49:44.037057 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.037025    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"]
Jul 23 22:49:44.039001 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:49:44.038972    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ce6f74_3fa4_4cae_ab70_0bd2c8a2bd58.slice/crio-dba930597f11eb056377230178c3f99160a1e9e38fd4e6dff4f2f2b5e72f6d40 WatchSource:0}: Error finding container dba930597f11eb056377230178c3f99160a1e9e38fd4e6dff4f2f2b5e72f6d40: Status 404 returned error can't find the container with id dba930597f11eb056377230178c3f99160a1e9e38fd4e6dff4f2f2b5e72f6d40
Jul 23 22:49:44.504597 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.504563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m" event={"ID":"c10c6c92-2a89-4235-982d-bc4c6035a695","Type":"ContainerStarted","Data":"cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843"}
Jul 23 22:49:44.504783 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.504613    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m" event={"ID":"c10c6c92-2a89-4235-982d-bc4c6035a695","Type":"ContainerStarted","Data":"3dd68cb6b17a63fdc239353fda910a47e8c6bd93672b3c5e93de7deb81d0ac3d"}
Jul 23 22:49:44.504783 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.504641    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:49:44.506250 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.506218    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn" event={"ID":"54b3f355-86af-4839-b5e4-5fb9f270ab92","Type":"ContainerStarted","Data":"6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11"}
Jul 23 22:49:44.506368 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.506267    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn" event={"ID":"54b3f355-86af-4839-b5e4-5fb9f270ab92","Type":"ContainerStarted","Data":"e38b154955376a07368f1d9c466dd6260e15278a431446acb7fc55672218753a"}
Jul 23 22:49:44.506368 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.506290    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:44.507776 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.507753    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg" event={"ID":"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58","Type":"ContainerStarted","Data":"661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0"}
Jul 23 22:49:44.507875 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.507790    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg" event={"ID":"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58","Type":"ContainerStarted","Data":"dba930597f11eb056377230178c3f99160a1e9e38fd4e6dff4f2f2b5e72f6d40"}
Jul 23 22:49:44.507875 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.507818    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:49:44.523438 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.523393    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m" podStartSLOduration=1.523377787 podStartE2EDuration="1.523377787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:49:44.522277372 +0000 UTC m=+1638.839806184" watchObservedRunningTime="2026-07-23 22:49:44.523377787 +0000 UTC m=+1638.840906574"
Jul 23 22:49:44.542108 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.542049    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn" podStartSLOduration=2.146461996 podStartE2EDuration="2.542029138s" totalImagesPullingTime="395.567142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:49:44.540050257 +0000 UTC m=+1638.857579066" watchObservedRunningTime="2026-07-23 22:49:44.542029138 +0000 UTC m=+1638.859557948"
Jul 23 22:49:44.556020 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:44.555867    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg" podStartSLOduration=1.55584458 podStartE2EDuration="1.55584458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:49:44.555192018 +0000 UTC m=+1638.872720829" watchObservedRunningTime="2026-07-23 22:49:44.55584458 +0000 UTC m=+1638.873373390"
Jul 23 22:49:50.518635 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:50.518601    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:49:55.514495 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:55.514457    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:49:55.514956 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:49:55.514734    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:50:18.202008 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.201958    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"]
Jul 23 22:50:18.202596 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.202175    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn" podUID="54b3f355-86af-4839-b5e4-5fb9f270ab92" containerName="maas-api" containerID="cri-o://6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11" gracePeriod=30
Jul 23 22:50:18.249136 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.249102    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"]
Jul 23 22:50:18.249397 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.249372    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg" podUID="e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58" containerName="payload-processing" containerID="cri-o://661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0" gracePeriod=30
Jul 23 22:50:18.256780 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.256745    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"]
Jul 23 22:50:18.257034 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.256997    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m" podUID="c10c6c92-2a89-4235-982d-bc4c6035a695" containerName="payload-pre-processing" containerID="cri-o://cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843" gracePeriod=30
Jul 23 22:50:18.539819 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.539791    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:50:18.564905 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.564876    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:50:18.568160 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.568133    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:50:18.581900 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.581871    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-plugins-config-volume\") pod \"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58\" (UID: \"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58\") "
Jul 23 22:50:18.582054 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.581909    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjppp\" (UniqueName: \"kubernetes.io/projected/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-kube-api-access-jjppp\") pod \"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58\" (UID: \"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58\") "
Jul 23 22:50:18.582255 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.582233    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58" (UID: "e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:50:18.584151 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.584043    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-kube-api-access-jjppp" (OuterVolumeSpecName: "kube-api-access-jjppp") pod "e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58" (UID: "e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58"). InnerVolumeSpecName "kube-api-access-jjppp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:50:18.668082 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.668048    2583 generic.go:356] "Generic (PLEG): container finished" podID="c10c6c92-2a89-4235-982d-bc4c6035a695" containerID="cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843" exitCode=0
Jul 23 22:50:18.668275 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.668090    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"
Jul 23 22:50:18.668275 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.668130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m" event={"ID":"c10c6c92-2a89-4235-982d-bc4c6035a695","Type":"ContainerDied","Data":"cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843"}
Jul 23 22:50:18.668275 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.668187    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m" event={"ID":"c10c6c92-2a89-4235-982d-bc4c6035a695","Type":"ContainerDied","Data":"3dd68cb6b17a63fdc239353fda910a47e8c6bd93672b3c5e93de7deb81d0ac3d"}
Jul 23 22:50:18.668275 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.668219    2583 scope.go:117] "RemoveContainer" containerID="cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843"
Jul 23 22:50:18.669515 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.669498    2583 generic.go:356] "Generic (PLEG): container finished" podID="54b3f355-86af-4839-b5e4-5fb9f270ab92" containerID="6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11" exitCode=0
Jul 23 22:50:18.669600 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.669538    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"
Jul 23 22:50:18.669600 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.669564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn" event={"ID":"54b3f355-86af-4839-b5e4-5fb9f270ab92","Type":"ContainerDied","Data":"6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11"}
Jul 23 22:50:18.669710 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.669610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn" event={"ID":"54b3f355-86af-4839-b5e4-5fb9f270ab92","Type":"ContainerDied","Data":"e38b154955376a07368f1d9c466dd6260e15278a431446acb7fc55672218753a"}
Jul 23 22:50:18.670949 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.670909    2583 generic.go:356] "Generic (PLEG): container finished" podID="e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58" containerID="661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0" exitCode=0
Jul 23 22:50:18.671059 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.670955    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg" event={"ID":"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58","Type":"ContainerDied","Data":"661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0"}
Jul 23 22:50:18.671059 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.670971    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"
Jul 23 22:50:18.671059 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.671003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg" event={"ID":"e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58","Type":"ContainerDied","Data":"dba930597f11eb056377230178c3f99160a1e9e38fd4e6dff4f2f2b5e72f6d40"}
Jul 23 22:50:18.677273 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.677255    2583 scope.go:117] "RemoveContainer" containerID="cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843"
Jul 23 22:50:18.677564 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:50:18.677539    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843\": container with ID starting with cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843 not found: ID does not exist" containerID="cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843"
Jul 23 22:50:18.677614 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.677575    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843"} err="failed to get container status \"cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843\": rpc error: code = NotFound desc = could not find container \"cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843\": container with ID starting with cc19c3bca8e48c9ff459c39a3bb1ad5102e7e87b9335d0efcc2c987495683843 not found: ID does not exist"
Jul 23 22:50:18.677614 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.677597    2583 scope.go:117] "RemoveContainer" containerID="6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11"
Jul 23 22:50:18.682834 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.682814    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10c6c92-2a89-4235-982d-bc4c6035a695-plugins-config-volume\") pod \"c10c6c92-2a89-4235-982d-bc4c6035a695\" (UID: \"c10c6c92-2a89-4235-982d-bc4c6035a695\") "
Jul 23 22:50:18.682965 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.682855    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8wzv\" (UniqueName: \"kubernetes.io/projected/c10c6c92-2a89-4235-982d-bc4c6035a695-kube-api-access-q8wzv\") pod \"c10c6c92-2a89-4235-982d-bc4c6035a695\" (UID: \"c10c6c92-2a89-4235-982d-bc4c6035a695\") "
Jul 23 22:50:18.682965 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.682906    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2xp4\" (UniqueName: \"kubernetes.io/projected/54b3f355-86af-4839-b5e4-5fb9f270ab92-kube-api-access-t2xp4\") pod \"54b3f355-86af-4839-b5e4-5fb9f270ab92\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") "
Jul 23 22:50:18.683075 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.682977    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b3f355-86af-4839-b5e4-5fb9f270ab92-openshift-service-ca\") pod \"54b3f355-86af-4839-b5e4-5fb9f270ab92\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") "
Jul 23 22:50:18.683075 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.683046    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b3f355-86af-4839-b5e4-5fb9f270ab92-maas-api-tls\") pod \"54b3f355-86af-4839-b5e4-5fb9f270ab92\" (UID: \"54b3f355-86af-4839-b5e4-5fb9f270ab92\") "
Jul 23 22:50:18.683354 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.683331    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:50:18.683461 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.683357    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjppp\" (UniqueName: \"kubernetes.io/projected/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58-kube-api-access-jjppp\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:50:18.683461 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.683286    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10c6c92-2a89-4235-982d-bc4c6035a695-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c10c6c92-2a89-4235-982d-bc4c6035a695" (UID: "c10c6c92-2a89-4235-982d-bc4c6035a695"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:50:18.683543 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.683466    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b3f355-86af-4839-b5e4-5fb9f270ab92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54b3f355-86af-4839-b5e4-5fb9f270ab92" (UID: "54b3f355-86af-4839-b5e4-5fb9f270ab92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:50:18.685130 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.685094    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b3f355-86af-4839-b5e4-5fb9f270ab92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54b3f355-86af-4839-b5e4-5fb9f270ab92" (UID: "54b3f355-86af-4839-b5e4-5fb9f270ab92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:50:18.685480 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.685453    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10c6c92-2a89-4235-982d-bc4c6035a695-kube-api-access-q8wzv" (OuterVolumeSpecName: "kube-api-access-q8wzv") pod "c10c6c92-2a89-4235-982d-bc4c6035a695" (UID: "c10c6c92-2a89-4235-982d-bc4c6035a695"). InnerVolumeSpecName "kube-api-access-q8wzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:50:18.685595 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.685490    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b3f355-86af-4839-b5e4-5fb9f270ab92-kube-api-access-t2xp4" (OuterVolumeSpecName: "kube-api-access-t2xp4") pod "54b3f355-86af-4839-b5e4-5fb9f270ab92" (UID: "54b3f355-86af-4839-b5e4-5fb9f270ab92"). InnerVolumeSpecName "kube-api-access-t2xp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:50:18.688422 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.687249    2583 scope.go:117] "RemoveContainer" containerID="6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11"
Jul 23 22:50:18.688422 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:50:18.688380    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11\": container with ID starting with 6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11 not found: ID does not exist" containerID="6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11"
Jul 23 22:50:18.688422 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.688407    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11"} err="failed to get container status \"6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11\": rpc error: code = NotFound desc = could not find container \"6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11\": container with ID starting with 6907568c886ad4033d6c4cf85eff7c52a47a9c08c9269ce686b88df035e95f11 not found: ID does not exist"
Jul 23 22:50:18.688583 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.688425    2583 scope.go:117] "RemoveContainer" containerID="661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0"
Jul 23 22:50:18.692651 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.692631    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"]
Jul 23 22:50:18.694314 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.694296    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9c6cb93-64644b7d8f-k29xg"]
Jul 23 22:50:18.696963 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.696948    2583 scope.go:117] "RemoveContainer" containerID="661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0"
Jul 23 22:50:18.697268 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:50:18.697248    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0\": container with ID starting with 661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0 not found: ID does not exist" containerID="661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0"
Jul 23 22:50:18.697327 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.697279    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0"} err="failed to get container status \"661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0\": rpc error: code = NotFound desc = could not find container \"661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0\": container with ID starting with 661f5259628691c2a40ac589681d7ccb50ed55a8b96b9f58defca68754e6f8f0 not found: ID does not exist"
Jul 23 22:50:18.784354 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.784265    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b3f355-86af-4839-b5e4-5fb9f270ab92-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:50:18.784354 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.784299    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54b3f355-86af-4839-b5e4-5fb9f270ab92-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:50:18.784354 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.784313    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10c6c92-2a89-4235-982d-bc4c6035a695-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:50:18.784354 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.784321    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8wzv\" (UniqueName: \"kubernetes.io/projected/c10c6c92-2a89-4235-982d-bc4c6035a695-kube-api-access-q8wzv\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:50:18.784354 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.784331    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2xp4\" (UniqueName: \"kubernetes.io/projected/54b3f355-86af-4839-b5e4-5fb9f270ab92-kube-api-access-t2xp4\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:50:18.992042 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.992011    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"]
Jul 23 22:50:18.995066 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:18.995038    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9c6cb93-84874c5dbd-cld5m"]
Jul 23 22:50:19.005586 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:19.005553    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"]
Jul 23 22:50:19.011906 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:19.011873    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9c6cb93-7fb4465774-bbcrn"]
Jul 23 22:50:20.228647 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:20.228610    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54b3f355-86af-4839-b5e4-5fb9f270ab92" path="/var/lib/kubelet/pods/54b3f355-86af-4839-b5e4-5fb9f270ab92/volumes"
Jul 23 22:50:20.229070 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:20.229019    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c10c6c92-2a89-4235-982d-bc4c6035a695" path="/var/lib/kubelet/pods/c10c6c92-2a89-4235-982d-bc4c6035a695/volumes"
Jul 23 22:50:20.229341 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:20.229328    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58" path="/var/lib/kubelet/pods/e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58/volumes"
Jul 23 22:50:26.026979 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.026890    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"]
Jul 23 22:50:26.027460 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.027410    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10c6c92-2a89-4235-982d-bc4c6035a695" containerName="payload-pre-processing"
Jul 23 22:50:26.027460 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.027429    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10c6c92-2a89-4235-982d-bc4c6035a695" containerName="payload-pre-processing"
Jul 23 22:50:26.027460 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.027445    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58" containerName="payload-processing"
Jul 23 22:50:26.027460 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.027456    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58" containerName="payload-processing"
Jul 23 22:50:26.027658 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.027489    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54b3f355-86af-4839-b5e4-5fb9f270ab92" containerName="maas-api"
Jul 23 22:50:26.027658 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.027498    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="54b3f355-86af-4839-b5e4-5fb9f270ab92" containerName="maas-api"
Jul 23 22:50:26.027658 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.027582    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="54b3f355-86af-4839-b5e4-5fb9f270ab92" containerName="maas-api"
Jul 23 22:50:26.027658 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.027596    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10c6c92-2a89-4235-982d-bc4c6035a695" containerName="payload-pre-processing"
Jul 23 22:50:26.027658 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.027609    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ce6f74-3fa4-4cae-ab70-0bd2c8a2bd58" containerName="payload-processing"
Jul 23 22:50:26.032548 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.032524    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.035254 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.035235    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 22:50:26.036103 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.036079    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 22:50:26.036233 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.036086    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ca65b203-serving-cert\""
Jul 23 22:50:26.036233 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.036089    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vw8vx\""
Jul 23 22:50:26.041136 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.040523    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"]
Jul 23 22:50:26.150875 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.150818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-maas-api-tls\") pod \"maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.151107 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.150882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc8z4\" (UniqueName: \"kubernetes.io/projected/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-kube-api-access-rc8z4\") pod \"maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.151107 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.150950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-openshift-service-ca\") pod \"maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.252375 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.252332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-maas-api-tls\") pod \"maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.252375 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.252374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc8z4\" (UniqueName: \"kubernetes.io/projected/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-kube-api-access-rc8z4\") pod \"maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.252620 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.252401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-openshift-service-ca\") pod \"maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.255178 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.255156    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ca65b203-serving-cert\""
Jul 23 22:50:26.255236 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.255184    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 22:50:26.260615 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.260592    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 22:50:26.262690 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:50:26.262671    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-serving-cert: secret "maas-api-e2e-ait-ca65b203-serving-cert" not found
Jul 23 22:50:26.262782 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:50:26.262745    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-maas-api-tls podName:403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6 nodeName:}" failed. No retries permitted until 2026-07-23 22:50:26.762726778 +0000 UTC m=+1681.080255566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-maas-api-tls") pod "maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l" (UID: "403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6") : secret "maas-api-e2e-ait-ca65b203-serving-cert" not found
Jul 23 22:50:26.263315 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.263289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-openshift-service-ca\") pod \"maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.271174 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.271144    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc8z4\" (UniqueName: \"kubernetes.io/projected/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-kube-api-access-rc8z4\") pod \"maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.711531 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.711490    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"]
Jul 23 22:50:26.716339 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.716316    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:50:26.719060 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.719034    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ca65b203\""
Jul 23 22:50:26.719210 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.719047    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ca65b203-dockercfg-kd5b8\""
Jul 23 22:50:26.724189 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.724163    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"]
Jul 23 22:50:26.769620 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.769581    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"]
Jul 23 22:50:26.773027 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.773007    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:50:26.783065 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.783040    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"]
Jul 23 22:50:26.859552 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.859514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9spd\" (UniqueName: \"kubernetes.io/projected/69c96063-38c8-45ce-ae4a-fd0b4131ae92-kube-api-access-g9spd\") pod \"payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm\" (UID: \"69c96063-38c8-45ce-ae4a-fd0b4131ae92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:50:26.859713 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.859571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-maas-api-tls\") pod \"maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.859713 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.859637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c96063-38c8-45ce-ae4a-fd0b4131ae92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm\" (UID: \"69c96063-38c8-45ce-ae4a-fd0b4131ae92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:50:26.859713 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.859672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk66n\" (UniqueName: \"kubernetes.io/projected/e27ab168-5e34-4609-81a7-8a3bb2938da7-kube-api-access-dk66n\") pod \"payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz\" (UID: \"e27ab168-5e34-4609-81a7-8a3bb2938da7\") " pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:50:26.859874 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.859775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27ab168-5e34-4609-81a7-8a3bb2938da7-plugins-config-volume\") pod \"payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz\" (UID: \"e27ab168-5e34-4609-81a7-8a3bb2938da7\") " pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:50:26.862117 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.862095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-maas-api-tls\") pod \"maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.947206 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.947164    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vw8vx\""
Jul 23 22:50:26.955219 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.955189    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:26.960479 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.960446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9spd\" (UniqueName: \"kubernetes.io/projected/69c96063-38c8-45ce-ae4a-fd0b4131ae92-kube-api-access-g9spd\") pod \"payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm\" (UID: \"69c96063-38c8-45ce-ae4a-fd0b4131ae92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:50:26.960593 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.960494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c96063-38c8-45ce-ae4a-fd0b4131ae92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm\" (UID: \"69c96063-38c8-45ce-ae4a-fd0b4131ae92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:50:26.960593 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.960518    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk66n\" (UniqueName: \"kubernetes.io/projected/e27ab168-5e34-4609-81a7-8a3bb2938da7-kube-api-access-dk66n\") pod \"payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz\" (UID: \"e27ab168-5e34-4609-81a7-8a3bb2938da7\") " pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:50:26.960593 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.960560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27ab168-5e34-4609-81a7-8a3bb2938da7-plugins-config-volume\") pod \"payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz\" (UID: \"e27ab168-5e34-4609-81a7-8a3bb2938da7\") " pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:50:26.961236 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.961213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c96063-38c8-45ce-ae4a-fd0b4131ae92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm\" (UID: \"69c96063-38c8-45ce-ae4a-fd0b4131ae92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:50:26.961347 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.961308    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27ab168-5e34-4609-81a7-8a3bb2938da7-plugins-config-volume\") pod \"payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz\" (UID: \"e27ab168-5e34-4609-81a7-8a3bb2938da7\") " pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:50:26.968752 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.968722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9spd\" (UniqueName: \"kubernetes.io/projected/69c96063-38c8-45ce-ae4a-fd0b4131ae92-kube-api-access-g9spd\") pod \"payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm\" (UID: \"69c96063-38c8-45ce-ae4a-fd0b4131ae92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:50:26.969038 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:26.969013    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk66n\" (UniqueName: \"kubernetes.io/projected/e27ab168-5e34-4609-81a7-8a3bb2938da7-kube-api-access-dk66n\") pod \"payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz\" (UID: \"e27ab168-5e34-4609-81a7-8a3bb2938da7\") " pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:50:27.028534 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.028503    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:50:27.084673 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.084618    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:50:27.106782 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.106739    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"]
Jul 23 22:50:27.109049 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:50:27.109016    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403a6ed1_b7d3_4b2f_920f_8d3dc08af9f6.slice/crio-3ecca071d861de55375fe79c10feb5eeabfb11f0ee59bd52e9700a04ed23b2ed WatchSource:0}: Error finding container 3ecca071d861de55375fe79c10feb5eeabfb11f0ee59bd52e9700a04ed23b2ed: Status 404 returned error can't find the container with id 3ecca071d861de55375fe79c10feb5eeabfb11f0ee59bd52e9700a04ed23b2ed
Jul 23 22:50:27.111004 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.110982    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 22:50:27.191463 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.191380    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"]
Jul 23 22:50:27.193127 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:50:27.193070    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c96063_38c8_45ce_ae4a_fd0b4131ae92.slice/crio-43aea59a65b6db3edc93f66c7f915d9d06987c07ec439d5d81908d0909b8b6e3 WatchSource:0}: Error finding container 43aea59a65b6db3edc93f66c7f915d9d06987c07ec439d5d81908d0909b8b6e3: Status 404 returned error can't find the container with id 43aea59a65b6db3edc93f66c7f915d9d06987c07ec439d5d81908d0909b8b6e3
Jul 23 22:50:27.244104 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.244075    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"]
Jul 23 22:50:27.246982 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:50:27.246954    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27ab168_5e34_4609_81a7_8a3bb2938da7.slice/crio-68e972445caf29528f4a71f96e8b797db72191825acbb763adf1f0cae0cc1c1c WatchSource:0}: Error finding container 68e972445caf29528f4a71f96e8b797db72191825acbb763adf1f0cae0cc1c1c: Status 404 returned error can't find the container with id 68e972445caf29528f4a71f96e8b797db72191825acbb763adf1f0cae0cc1c1c
Jul 23 22:50:27.721156 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.721116    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l" event={"ID":"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6","Type":"ContainerStarted","Data":"37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc"}
Jul 23 22:50:27.721351 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.721211    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:27.721351 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.721234    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l" event={"ID":"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6","Type":"ContainerStarted","Data":"3ecca071d861de55375fe79c10feb5eeabfb11f0ee59bd52e9700a04ed23b2ed"}
Jul 23 22:50:27.722810 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.722781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm" event={"ID":"69c96063-38c8-45ce-ae4a-fd0b4131ae92","Type":"ContainerStarted","Data":"329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0"}
Jul 23 22:50:27.722964 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.722824    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm" event={"ID":"69c96063-38c8-45ce-ae4a-fd0b4131ae92","Type":"ContainerStarted","Data":"43aea59a65b6db3edc93f66c7f915d9d06987c07ec439d5d81908d0909b8b6e3"}
Jul 23 22:50:27.722964 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.722896    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:50:27.724462 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.724437    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz" event={"ID":"e27ab168-5e34-4609-81a7-8a3bb2938da7","Type":"ContainerStarted","Data":"f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff"}
Jul 23 22:50:27.724573 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.724479    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz" event={"ID":"e27ab168-5e34-4609-81a7-8a3bb2938da7","Type":"ContainerStarted","Data":"68e972445caf29528f4a71f96e8b797db72191825acbb763adf1f0cae0cc1c1c"}
Jul 23 22:50:27.724573 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.724530    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:50:27.739441 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.739377    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l" podStartSLOduration=1.273346129 podStartE2EDuration="1.739357121s" totalImagesPullingTime="466.010992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:50:27.737162029 +0000 UTC m=+1682.054690862" watchObservedRunningTime="2026-07-23 22:50:27.739357121 +0000 UTC m=+1682.056885932"
Jul 23 22:50:27.754478 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.754415    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm" podStartSLOduration=1.754397174 podStartE2EDuration="1.754397174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:50:27.752776936 +0000 UTC m=+1682.070305744" watchObservedRunningTime="2026-07-23 22:50:27.754397174 +0000 UTC m=+1682.071925985"
Jul 23 22:50:27.772711 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:27.772654    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz" podStartSLOduration=1.77263738 podStartE2EDuration="1.77263738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:50:27.771788809 +0000 UTC m=+1682.089317628" watchObservedRunningTime="2026-07-23 22:50:27.77263738 +0000 UTC m=+1682.090166189"
Jul 23 22:50:33.735147 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:33.735116    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:50:38.731123 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:38.731091    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:50:38.731539 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:50:38.731375    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:51:02.177902 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.177813    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"]
Jul 23 22:51:02.178451 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.178086    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l" podUID="403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6" containerName="maas-api" containerID="cri-o://37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc" gracePeriod=30
Jul 23 22:51:02.236490 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.236455    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"]
Jul 23 22:51:02.236698 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.236673    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz" podUID="e27ab168-5e34-4609-81a7-8a3bb2938da7" containerName="payload-processing" containerID="cri-o://f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff" gracePeriod=30
Jul 23 22:51:02.244542 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.244508    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"]
Jul 23 22:51:02.244791 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.244741    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm" podUID="69c96063-38c8-45ce-ae4a-fd0b4131ae92" containerName="payload-pre-processing" containerID="cri-o://329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0" gracePeriod=30
Jul 23 22:51:02.526127 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.526088    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:51:02.561562 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.561534    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:51:02.567174 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.567151    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:51:02.575527 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.575505    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc8z4\" (UniqueName: \"kubernetes.io/projected/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-kube-api-access-rc8z4\") pod \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") "
Jul 23 22:51:02.575709 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.575584    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-openshift-service-ca\") pod \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") "
Jul 23 22:51:02.575709 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.575627    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-maas-api-tls\") pod \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\" (UID: \"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6\") "
Jul 23 22:51:02.575945 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.575904    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6" (UID: "403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:51:02.577678 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.577652    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-kube-api-access-rc8z4" (OuterVolumeSpecName: "kube-api-access-rc8z4") pod "403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6" (UID: "403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6"). InnerVolumeSpecName "kube-api-access-rc8z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:51:02.577764 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.577700    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6" (UID: "403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:51:02.676372 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.676335    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9spd\" (UniqueName: \"kubernetes.io/projected/69c96063-38c8-45ce-ae4a-fd0b4131ae92-kube-api-access-g9spd\") pod \"69c96063-38c8-45ce-ae4a-fd0b4131ae92\" (UID: \"69c96063-38c8-45ce-ae4a-fd0b4131ae92\") "
Jul 23 22:51:02.676372 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.676377    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk66n\" (UniqueName: \"kubernetes.io/projected/e27ab168-5e34-4609-81a7-8a3bb2938da7-kube-api-access-dk66n\") pod \"e27ab168-5e34-4609-81a7-8a3bb2938da7\" (UID: \"e27ab168-5e34-4609-81a7-8a3bb2938da7\") "
Jul 23 22:51:02.676626 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.676400    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c96063-38c8-45ce-ae4a-fd0b4131ae92-plugins-config-volume\") pod \"69c96063-38c8-45ce-ae4a-fd0b4131ae92\" (UID: \"69c96063-38c8-45ce-ae4a-fd0b4131ae92\") "
Jul 23 22:51:02.676626 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.676436    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27ab168-5e34-4609-81a7-8a3bb2938da7-plugins-config-volume\") pod \"e27ab168-5e34-4609-81a7-8a3bb2938da7\" (UID: \"e27ab168-5e34-4609-81a7-8a3bb2938da7\") "
Jul 23 22:51:02.676626 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.676607    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc8z4\" (UniqueName: \"kubernetes.io/projected/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-kube-api-access-rc8z4\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:02.676789 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.676643    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:02.676789 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.676658    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:02.676875 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.676807    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c96063-38c8-45ce-ae4a-fd0b4131ae92-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69c96063-38c8-45ce-ae4a-fd0b4131ae92" (UID: "69c96063-38c8-45ce-ae4a-fd0b4131ae92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:51:02.676875 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.676847    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27ab168-5e34-4609-81a7-8a3bb2938da7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e27ab168-5e34-4609-81a7-8a3bb2938da7" (UID: "e27ab168-5e34-4609-81a7-8a3bb2938da7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:51:02.678634 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.678610    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c96063-38c8-45ce-ae4a-fd0b4131ae92-kube-api-access-g9spd" (OuterVolumeSpecName: "kube-api-access-g9spd") pod "69c96063-38c8-45ce-ae4a-fd0b4131ae92" (UID: "69c96063-38c8-45ce-ae4a-fd0b4131ae92"). InnerVolumeSpecName "kube-api-access-g9spd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:51:02.678705 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.678645    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27ab168-5e34-4609-81a7-8a3bb2938da7-kube-api-access-dk66n" (OuterVolumeSpecName: "kube-api-access-dk66n") pod "e27ab168-5e34-4609-81a7-8a3bb2938da7" (UID: "e27ab168-5e34-4609-81a7-8a3bb2938da7"). InnerVolumeSpecName "kube-api-access-dk66n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:51:02.777364 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.777266    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e27ab168-5e34-4609-81a7-8a3bb2938da7-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:02.777364 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.777300    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9spd\" (UniqueName: \"kubernetes.io/projected/69c96063-38c8-45ce-ae4a-fd0b4131ae92-kube-api-access-g9spd\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:02.777364 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.777312    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk66n\" (UniqueName: \"kubernetes.io/projected/e27ab168-5e34-4609-81a7-8a3bb2938da7-kube-api-access-dk66n\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:02.777364 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.777323    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c96063-38c8-45ce-ae4a-fd0b4131ae92-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:02.902335 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.902294    2583 generic.go:356] "Generic (PLEG): container finished" podID="69c96063-38c8-45ce-ae4a-fd0b4131ae92" containerID="329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0" exitCode=0
Jul 23 22:51:02.902532 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.902344    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"
Jul 23 22:51:02.902532 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.902383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm" event={"ID":"69c96063-38c8-45ce-ae4a-fd0b4131ae92","Type":"ContainerDied","Data":"329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0"}
Jul 23 22:51:02.902532 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.902448    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm" event={"ID":"69c96063-38c8-45ce-ae4a-fd0b4131ae92","Type":"ContainerDied","Data":"43aea59a65b6db3edc93f66c7f915d9d06987c07ec439d5d81908d0909b8b6e3"}
Jul 23 22:51:02.902532 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.902478    2583 scope.go:117] "RemoveContainer" containerID="329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0"
Jul 23 22:51:02.903750 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.903728    2583 generic.go:356] "Generic (PLEG): container finished" podID="e27ab168-5e34-4609-81a7-8a3bb2938da7" containerID="f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff" exitCode=0
Jul 23 22:51:02.903856 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.903777    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"
Jul 23 22:51:02.903856 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.903807    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz" event={"ID":"e27ab168-5e34-4609-81a7-8a3bb2938da7","Type":"ContainerDied","Data":"f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff"}
Jul 23 22:51:02.903992 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.903859    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz" event={"ID":"e27ab168-5e34-4609-81a7-8a3bb2938da7","Type":"ContainerDied","Data":"68e972445caf29528f4a71f96e8b797db72191825acbb763adf1f0cae0cc1c1c"}
Jul 23 22:51:02.909973 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.905646    2583 generic.go:356] "Generic (PLEG): container finished" podID="403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6" containerID="37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc" exitCode=0
Jul 23 22:51:02.909973 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.905695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l" event={"ID":"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6","Type":"ContainerDied","Data":"37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc"}
Jul 23 22:51:02.909973 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.905725    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"
Jul 23 22:51:02.909973 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.905780    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l" event={"ID":"403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6","Type":"ContainerDied","Data":"3ecca071d861de55375fe79c10feb5eeabfb11f0ee59bd52e9700a04ed23b2ed"}
Jul 23 22:51:02.919638 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.919615    2583 scope.go:117] "RemoveContainer" containerID="329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0"
Jul 23 22:51:02.919904 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:51:02.919884    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0\": container with ID starting with 329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0 not found: ID does not exist" containerID="329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0"
Jul 23 22:51:02.919973 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.919916    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0"} err="failed to get container status \"329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0\": rpc error: code = NotFound desc = could not find container \"329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0\": container with ID starting with 329ea122361e47297b804aaa241f1c67e50926ce74faa24e3af169c032f2b3d0 not found: ID does not exist"
Jul 23 22:51:02.919973 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.919965    2583 scope.go:117] "RemoveContainer" containerID="f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff"
Jul 23 22:51:02.928768 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.928730    2583 scope.go:117] "RemoveContainer" containerID="f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff"
Jul 23 22:51:02.929115 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:51:02.929094    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff\": container with ID starting with f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff not found: ID does not exist" containerID="f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff"
Jul 23 22:51:02.929229 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.929125    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff"} err="failed to get container status \"f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff\": rpc error: code = NotFound desc = could not find container \"f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff\": container with ID starting with f9b36d26e91b5721d8eacd4620650807d791e0df0ea430a207056ade4d4d17ff not found: ID does not exist"
Jul 23 22:51:02.929229 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.929150    2583 scope.go:117] "RemoveContainer" containerID="37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc"
Jul 23 22:51:02.931399 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.931377    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"]
Jul 23 22:51:02.935525 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.935504    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca65b203-5f78db946d-2snsm"]
Jul 23 22:51:02.937410 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.937394    2583 scope.go:117] "RemoveContainer" containerID="37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc"
Jul 23 22:51:02.937681 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:51:02.937661    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc\": container with ID starting with 37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc not found: ID does not exist" containerID="37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc"
Jul 23 22:51:02.937738 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.937688    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc"} err="failed to get container status \"37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc\": rpc error: code = NotFound desc = could not find container \"37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc\": container with ID starting with 37427e20638a72fc67aa1902aad9dbd396c5586985a681bc555e8c4d6bf57ecc not found: ID does not exist"
Jul 23 22:51:02.947177 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.947152    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"]
Jul 23 22:51:02.951497 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.951475    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca65b203-6f9d665999-r7f4l"]
Jul 23 22:51:02.965351 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.965330    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"]
Jul 23 22:51:02.970220 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:02.970197    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca65b203-57bbc74876-rtsnz"]
Jul 23 22:51:04.228465 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:04.228430    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6" path="/var/lib/kubelet/pods/403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6/volumes"
Jul 23 22:51:04.228859 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:04.228795    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69c96063-38c8-45ce-ae4a-fd0b4131ae92" path="/var/lib/kubelet/pods/69c96063-38c8-45ce-ae4a-fd0b4131ae92/volumes"
Jul 23 22:51:04.229156 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:04.229142    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e27ab168-5e34-4609-81a7-8a3bb2938da7" path="/var/lib/kubelet/pods/e27ab168-5e34-4609-81a7-8a3bb2938da7/volumes"
Jul 23 22:51:11.291907 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.291865    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"]
Jul 23 22:51:11.292511 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.292358    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27ab168-5e34-4609-81a7-8a3bb2938da7" containerName="payload-processing"
Jul 23 22:51:11.292511 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.292379    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27ab168-5e34-4609-81a7-8a3bb2938da7" containerName="payload-processing"
Jul 23 22:51:11.292511 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.292395    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c96063-38c8-45ce-ae4a-fd0b4131ae92" containerName="payload-pre-processing"
Jul 23 22:51:11.292511 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.292404    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c96063-38c8-45ce-ae4a-fd0b4131ae92" containerName="payload-pre-processing"
Jul 23 22:51:11.292511 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.292429    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6" containerName="maas-api"
Jul 23 22:51:11.292511 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.292439    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6" containerName="maas-api"
Jul 23 22:51:11.292813 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.292543    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="403a6ed1-b7d3-4b2f-920f-8d3dc08af9f6" containerName="maas-api"
Jul 23 22:51:11.292813 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.292557    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e27ab168-5e34-4609-81a7-8a3bb2938da7" containerName="payload-processing"
Jul 23 22:51:11.292813 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.292571    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c96063-38c8-45ce-ae4a-fd0b4131ae92" containerName="payload-pre-processing"
Jul 23 22:51:11.296816 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.296794    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:11.299684 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.299662    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 22:51:11.299684 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.299674    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vw8vx\""
Jul 23 22:51:11.300503 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.300483    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-cd5c7e7d-serving-cert\""
Jul 23 22:51:11.300596 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.300500    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 22:51:11.304889 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.304609    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"]
Jul 23 22:51:11.350550 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.350521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-openshift-service-ca\") pod \"maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:11.350678 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.350569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2g6w\" (UniqueName: \"kubernetes.io/projected/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-kube-api-access-w2g6w\") pod \"maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:11.350727 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.350684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-maas-api-tls\") pod \"maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:11.452072 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.452037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-maas-api-tls\") pod \"maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:11.452072 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.452079    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-openshift-service-ca\") pod \"maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:11.452312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.452113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2g6w\" (UniqueName: \"kubernetes.io/projected/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-kube-api-access-w2g6w\") pod \"maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:11.452312 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:51:11.452187    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-serving-cert: secret "maas-api-e2e-derive-cd5c7e7d-serving-cert" not found
Jul 23 22:51:11.452312 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:51:11.452260    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-maas-api-tls podName:03b032ee-c6d9-4f02-9fea-fe46ea236bf6 nodeName:}" failed. No retries permitted until 2026-07-23 22:51:11.95224448 +0000 UTC m=+1726.269773272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-maas-api-tls") pod "maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq" (UID: "03b032ee-c6d9-4f02-9fea-fe46ea236bf6") : secret "maas-api-e2e-derive-cd5c7e7d-serving-cert" not found
Jul 23 22:51:11.452654 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.452633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-openshift-service-ca\") pod \"maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:11.460680 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.460662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2g6w\" (UniqueName: \"kubernetes.io/projected/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-kube-api-access-w2g6w\") pod \"maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:11.888039 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.887999    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"]
Jul 23 22:51:11.891679 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.891655    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:11.894241 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.894217    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-cd5c7e7d\""
Jul 23 22:51:11.894359 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.894279    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-cd5c7e7d-dockercfg-ks69f\""
Jul 23 22:51:11.896904 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.896881    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"]
Jul 23 22:51:11.900287 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.900265    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:11.903797 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.903772    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"]
Jul 23 22:51:11.909396 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.909374    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"]
Jul 23 22:51:11.955280 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.955241    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw22k\" (UniqueName: \"kubernetes.io/projected/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-kube-api-access-kw22k\") pod \"payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57\" (UID: \"976e0f61-a6fc-42f5-b96c-25321b4ec9c5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:11.955443 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.955305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57\" (UID: \"976e0f61-a6fc-42f5-b96c-25321b4ec9c5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:11.955443 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.955327    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3762f63-79a6-4937-8d2a-52fe7490cafc-plugins-config-volume\") pod \"payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs\" (UID: \"c3762f63-79a6-4937-8d2a-52fe7490cafc\") " pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:11.955443 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.955364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8tdv\" (UniqueName: \"kubernetes.io/projected/c3762f63-79a6-4937-8d2a-52fe7490cafc-kube-api-access-r8tdv\") pod \"payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs\" (UID: \"c3762f63-79a6-4937-8d2a-52fe7490cafc\") " pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:11.955550 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.955436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-maas-api-tls\") pod \"maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:11.957809 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:11.957781    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-maas-api-tls\") pod \"maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:12.056920 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.056884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw22k\" (UniqueName: \"kubernetes.io/projected/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-kube-api-access-kw22k\") pod \"payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57\" (UID: \"976e0f61-a6fc-42f5-b96c-25321b4ec9c5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:12.057121 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.056977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57\" (UID: \"976e0f61-a6fc-42f5-b96c-25321b4ec9c5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:12.057121 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.057001    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3762f63-79a6-4937-8d2a-52fe7490cafc-plugins-config-volume\") pod \"payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs\" (UID: \"c3762f63-79a6-4937-8d2a-52fe7490cafc\") " pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:12.057121 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.057034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tdv\" (UniqueName: \"kubernetes.io/projected/c3762f63-79a6-4937-8d2a-52fe7490cafc-kube-api-access-r8tdv\") pod \"payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs\" (UID: \"c3762f63-79a6-4937-8d2a-52fe7490cafc\") " pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:12.057773 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.057745    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3762f63-79a6-4937-8d2a-52fe7490cafc-plugins-config-volume\") pod \"payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs\" (UID: \"c3762f63-79a6-4937-8d2a-52fe7490cafc\") " pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:12.057868 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.057746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57\" (UID: \"976e0f61-a6fc-42f5-b96c-25321b4ec9c5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:12.065522 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.065499    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tdv\" (UniqueName: \"kubernetes.io/projected/c3762f63-79a6-4937-8d2a-52fe7490cafc-kube-api-access-r8tdv\") pod \"payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs\" (UID: \"c3762f63-79a6-4937-8d2a-52fe7490cafc\") " pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:12.065741 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.065718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw22k\" (UniqueName: \"kubernetes.io/projected/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-kube-api-access-kw22k\") pod \"payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57\" (UID: \"976e0f61-a6fc-42f5-b96c-25321b4ec9c5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:12.203738 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.203639    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:12.208890 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.208865    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:12.215038 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.215011    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:12.369582 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.369493    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"]
Jul 23 22:51:12.371936 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:51:12.371896    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976e0f61_a6fc_42f5_b96c_25321b4ec9c5.slice/crio-0c2cc6627d2dc343e7bfb4b6c2b0c0b744fbf6173df3bfefe65c491afa4a7fb5 WatchSource:0}: Error finding container 0c2cc6627d2dc343e7bfb4b6c2b0c0b744fbf6173df3bfefe65c491afa4a7fb5: Status 404 returned error can't find the container with id 0c2cc6627d2dc343e7bfb4b6c2b0c0b744fbf6173df3bfefe65c491afa4a7fb5
Jul 23 22:51:12.596862 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.596836    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"]
Jul 23 22:51:12.599551 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:51:12.599521    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3762f63_79a6_4937_8d2a_52fe7490cafc.slice/crio-10aa7763a49db36284d8c60cb63096a43f41a41850d3454bc1537a2f62b0dbc1 WatchSource:0}: Error finding container 10aa7763a49db36284d8c60cb63096a43f41a41850d3454bc1537a2f62b0dbc1: Status 404 returned error can't find the container with id 10aa7763a49db36284d8c60cb63096a43f41a41850d3454bc1537a2f62b0dbc1
Jul 23 22:51:12.600795 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.600773    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"]
Jul 23 22:51:12.603182 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:51:12.603159    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b032ee_c6d9_4f02_9fea_fe46ea236bf6.slice/crio-a9416a70eebf899f3ce66941d7597d831a4d8c48cbe61131fc42bba339e647cb WatchSource:0}: Error finding container a9416a70eebf899f3ce66941d7597d831a4d8c48cbe61131fc42bba339e647cb: Status 404 returned error can't find the container with id a9416a70eebf899f3ce66941d7597d831a4d8c48cbe61131fc42bba339e647cb
Jul 23 22:51:12.962468 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.962438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq" event={"ID":"03b032ee-c6d9-4f02-9fea-fe46ea236bf6","Type":"ContainerStarted","Data":"a9416a70eebf899f3ce66941d7597d831a4d8c48cbe61131fc42bba339e647cb"}
Jul 23 22:51:12.964110 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.964083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57" event={"ID":"976e0f61-a6fc-42f5-b96c-25321b4ec9c5","Type":"ContainerStarted","Data":"cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df"}
Jul 23 22:51:12.964219 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.964136    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57" event={"ID":"976e0f61-a6fc-42f5-b96c-25321b4ec9c5","Type":"ContainerStarted","Data":"0c2cc6627d2dc343e7bfb4b6c2b0c0b744fbf6173df3bfefe65c491afa4a7fb5"}
Jul 23 22:51:12.964219 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.964167    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:12.965725 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.965699    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs" event={"ID":"c3762f63-79a6-4937-8d2a-52fe7490cafc","Type":"ContainerStarted","Data":"c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6"}
Jul 23 22:51:12.965795 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.965748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs" event={"ID":"c3762f63-79a6-4937-8d2a-52fe7490cafc","Type":"ContainerStarted","Data":"10aa7763a49db36284d8c60cb63096a43f41a41850d3454bc1537a2f62b0dbc1"}
Jul 23 22:51:12.965795 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.965768    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:12.981322 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.981278    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57" podStartSLOduration=1.981263312 podStartE2EDuration="1.981263312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:51:12.980013676 +0000 UTC m=+1727.297542487" watchObservedRunningTime="2026-07-23 22:51:12.981263312 +0000 UTC m=+1727.298792120"
Jul 23 22:51:12.997864 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:12.997814    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs" podStartSLOduration=1.997792933 podStartE2EDuration="1.997792933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:51:12.996529168 +0000 UTC m=+1727.314057980" watchObservedRunningTime="2026-07-23 22:51:12.997792933 +0000 UTC m=+1727.315321745"
Jul 23 22:51:13.972870 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:13.972821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq" event={"ID":"03b032ee-c6d9-4f02-9fea-fe46ea236bf6","Type":"ContainerStarted","Data":"f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff"}
Jul 23 22:51:13.973311 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:13.973059    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:13.990148 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:13.990095    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq" podStartSLOduration=2.638183602 podStartE2EDuration="2.990078476s" totalImagesPullingTime="351.894874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:51:13.988644793 +0000 UTC m=+1728.306173602" watchObservedRunningTime="2026-07-23 22:51:13.990078476 +0000 UTC m=+1728.307607286"
Jul 23 22:51:19.985160 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:19.985123    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:23.975052 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:23.975016    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:23.975464 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:23.975100    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:46.375441 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.375409    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"]
Jul 23 22:51:46.375912 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.375600    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq" podUID="03b032ee-c6d9-4f02-9fea-fe46ea236bf6" containerName="maas-api" containerID="cri-o://f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff" gracePeriod=30
Jul 23 22:51:46.413110 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.413076    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"]
Jul 23 22:51:46.413320 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.413280    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs" podUID="c3762f63-79a6-4937-8d2a-52fe7490cafc" containerName="payload-processing" containerID="cri-o://c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6" gracePeriod=30
Jul 23 22:51:46.420739 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.420711    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"]
Jul 23 22:51:46.420954 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.420913    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57" podUID="976e0f61-a6fc-42f5-b96c-25321b4ec9c5" containerName="payload-pre-processing" containerID="cri-o://cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df" gracePeriod=30
Jul 23 22:51:46.674960 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.674913    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:46.719145 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.719116    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:46.722007 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.721983    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:46.776695 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.776648    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3762f63-79a6-4937-8d2a-52fe7490cafc-plugins-config-volume\") pod \"c3762f63-79a6-4937-8d2a-52fe7490cafc\" (UID: \"c3762f63-79a6-4937-8d2a-52fe7490cafc\") "
Jul 23 22:51:46.776695 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.776699    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8tdv\" (UniqueName: \"kubernetes.io/projected/c3762f63-79a6-4937-8d2a-52fe7490cafc-kube-api-access-r8tdv\") pod \"c3762f63-79a6-4937-8d2a-52fe7490cafc\" (UID: \"c3762f63-79a6-4937-8d2a-52fe7490cafc\") "
Jul 23 22:51:46.777002 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.776735    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2g6w\" (UniqueName: \"kubernetes.io/projected/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-kube-api-access-w2g6w\") pod \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") "
Jul 23 22:51:46.777002 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.776756    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-maas-api-tls\") pod \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") "
Jul 23 22:51:46.777002 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.776775    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-openshift-service-ca\") pod \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\" (UID: \"03b032ee-c6d9-4f02-9fea-fe46ea236bf6\") "
Jul 23 22:51:46.777002 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.776818    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-plugins-config-volume\") pod \"976e0f61-a6fc-42f5-b96c-25321b4ec9c5\" (UID: \"976e0f61-a6fc-42f5-b96c-25321b4ec9c5\") "
Jul 23 22:51:46.777233 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.777006    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw22k\" (UniqueName: \"kubernetes.io/projected/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-kube-api-access-kw22k\") pod \"976e0f61-a6fc-42f5-b96c-25321b4ec9c5\" (UID: \"976e0f61-a6fc-42f5-b96c-25321b4ec9c5\") "
Jul 23 22:51:46.777233 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.777101    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3762f63-79a6-4937-8d2a-52fe7490cafc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3762f63-79a6-4937-8d2a-52fe7490cafc" (UID: "c3762f63-79a6-4937-8d2a-52fe7490cafc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:51:46.777376 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.777347    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "976e0f61-a6fc-42f5-b96c-25321b4ec9c5" (UID: "976e0f61-a6fc-42f5-b96c-25321b4ec9c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:51:46.777441 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.777355    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3762f63-79a6-4937-8d2a-52fe7490cafc-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:46.777499 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.777475    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03b032ee-c6d9-4f02-9fea-fe46ea236bf6" (UID: "03b032ee-c6d9-4f02-9fea-fe46ea236bf6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:51:46.779151 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.779103    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-kube-api-access-kw22k" (OuterVolumeSpecName: "kube-api-access-kw22k") pod "976e0f61-a6fc-42f5-b96c-25321b4ec9c5" (UID: "976e0f61-a6fc-42f5-b96c-25321b4ec9c5"). InnerVolumeSpecName "kube-api-access-kw22k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:51:46.779151 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.779136    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3762f63-79a6-4937-8d2a-52fe7490cafc-kube-api-access-r8tdv" (OuterVolumeSpecName: "kube-api-access-r8tdv") pod "c3762f63-79a6-4937-8d2a-52fe7490cafc" (UID: "c3762f63-79a6-4937-8d2a-52fe7490cafc"). InnerVolumeSpecName "kube-api-access-r8tdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:51:46.779479 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.779458    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-kube-api-access-w2g6w" (OuterVolumeSpecName: "kube-api-access-w2g6w") pod "03b032ee-c6d9-4f02-9fea-fe46ea236bf6" (UID: "03b032ee-c6d9-4f02-9fea-fe46ea236bf6"). InnerVolumeSpecName "kube-api-access-w2g6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:51:46.779523 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.779492    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03b032ee-c6d9-4f02-9fea-fe46ea236bf6" (UID: "03b032ee-c6d9-4f02-9fea-fe46ea236bf6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:51:46.877981 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.877913    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw22k\" (UniqueName: \"kubernetes.io/projected/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-kube-api-access-kw22k\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:46.877981 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.877979    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8tdv\" (UniqueName: \"kubernetes.io/projected/c3762f63-79a6-4937-8d2a-52fe7490cafc-kube-api-access-r8tdv\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:46.878212 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.877999    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2g6w\" (UniqueName: \"kubernetes.io/projected/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-kube-api-access-w2g6w\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:46.878212 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.878014    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:46.878212 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.878030    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b032ee-c6d9-4f02-9fea-fe46ea236bf6-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:46.878212 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:46.878045    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976e0f61-a6fc-42f5-b96c-25321b4ec9c5-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:51:47.131836 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.131799    2583 generic.go:356] "Generic (PLEG): container finished" podID="976e0f61-a6fc-42f5-b96c-25321b4ec9c5" containerID="cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df" exitCode=0
Jul 23 22:51:47.131836 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.131837    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"
Jul 23 22:51:47.131836 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.131886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57" event={"ID":"976e0f61-a6fc-42f5-b96c-25321b4ec9c5","Type":"ContainerDied","Data":"cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df"}
Jul 23 22:51:47.132215 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.131964    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57" event={"ID":"976e0f61-a6fc-42f5-b96c-25321b4ec9c5","Type":"ContainerDied","Data":"0c2cc6627d2dc343e7bfb4b6c2b0c0b744fbf6173df3bfefe65c491afa4a7fb5"}
Jul 23 22:51:47.132215 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.131988    2583 scope.go:117] "RemoveContainer" containerID="cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df"
Jul 23 22:51:47.133184 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.133162    2583 generic.go:356] "Generic (PLEG): container finished" podID="c3762f63-79a6-4937-8d2a-52fe7490cafc" containerID="c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6" exitCode=0
Jul 23 22:51:47.133260 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.133207    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"
Jul 23 22:51:47.133317 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.133251    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs" event={"ID":"c3762f63-79a6-4937-8d2a-52fe7490cafc","Type":"ContainerDied","Data":"c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6"}
Jul 23 22:51:47.133317 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.133295    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs" event={"ID":"c3762f63-79a6-4937-8d2a-52fe7490cafc","Type":"ContainerDied","Data":"10aa7763a49db36284d8c60cb63096a43f41a41850d3454bc1537a2f62b0dbc1"}
Jul 23 22:51:47.137725 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.137700    2583 generic.go:356] "Generic (PLEG): container finished" podID="03b032ee-c6d9-4f02-9fea-fe46ea236bf6" containerID="f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff" exitCode=0
Jul 23 22:51:47.137841 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.137771    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq" event={"ID":"03b032ee-c6d9-4f02-9fea-fe46ea236bf6","Type":"ContainerDied","Data":"f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff"}
Jul 23 22:51:47.137916 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.137850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq" event={"ID":"03b032ee-c6d9-4f02-9fea-fe46ea236bf6","Type":"ContainerDied","Data":"a9416a70eebf899f3ce66941d7597d831a4d8c48cbe61131fc42bba339e647cb"}
Jul 23 22:51:47.138000 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.137970    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"
Jul 23 22:51:47.146560 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.146541    2583 scope.go:117] "RemoveContainer" containerID="cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df"
Jul 23 22:51:47.146855 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:51:47.146830    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df\": container with ID starting with cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df not found: ID does not exist" containerID="cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df"
Jul 23 22:51:47.147020 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.146859    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df"} err="failed to get container status \"cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df\": rpc error: code = NotFound desc = could not find container \"cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df\": container with ID starting with cfb94ea63ad659e936192d98f69cd089502f1377c1d16e50e6bd91d41e21b8df not found: ID does not exist"
Jul 23 22:51:47.147020 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.146874    2583 scope.go:117] "RemoveContainer" containerID="c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6"
Jul 23 22:51:47.158398 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.158374    2583 scope.go:117] "RemoveContainer" containerID="c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6"
Jul 23 22:51:47.159672 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:51:47.159142    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6\": container with ID starting with c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6 not found: ID does not exist" containerID="c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6"
Jul 23 22:51:47.159672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.159198    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6"} err="failed to get container status \"c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6\": rpc error: code = NotFound desc = could not find container \"c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6\": container with ID starting with c91df7e2d5aaf42b72da661826fb8ee833524461ba6b155426dc9623804574a6 not found: ID does not exist"
Jul 23 22:51:47.159672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.159222    2583 scope.go:117] "RemoveContainer" containerID="f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff"
Jul 23 22:51:47.160268 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.160248    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"]
Jul 23 22:51:47.161937 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.161906    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cd5c7e7d-7cfb98f64d-6vk57"]
Jul 23 22:51:47.166837 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.166815    2583 scope.go:117] "RemoveContainer" containerID="f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff"
Jul 23 22:51:47.167099 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:51:47.167083    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff\": container with ID starting with f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff not found: ID does not exist" containerID="f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff"
Jul 23 22:51:47.167160 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.167105    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff"} err="failed to get container status \"f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff\": rpc error: code = NotFound desc = could not find container \"f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff\": container with ID starting with f35098ad8ff2ea430e63956b571ca2a623299ac6ac9fb274cbcc11357e68bbff not found: ID does not exist"
Jul 23 22:51:47.172684 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.172660    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"]
Jul 23 22:51:47.176485 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.176464    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd5c7e7d-674ff4f788-j9kdq"]
Jul 23 22:51:47.190373 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.190352    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"]
Jul 23 22:51:47.196250 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:47.196231    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cd5c7e7d-847748bf4b-gm6bs"]
Jul 23 22:51:48.228609 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:48.228580    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b032ee-c6d9-4f02-9fea-fe46ea236bf6" path="/var/lib/kubelet/pods/03b032ee-c6d9-4f02-9fea-fe46ea236bf6/volumes"
Jul 23 22:51:48.228997 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:48.228974    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976e0f61-a6fc-42f5-b96c-25321b4ec9c5" path="/var/lib/kubelet/pods/976e0f61-a6fc-42f5-b96c-25321b4ec9c5/volumes"
Jul 23 22:51:48.229328 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:51:48.229314    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3762f63-79a6-4937-8d2a-52fe7490cafc" path="/var/lib/kubelet/pods/c3762f63-79a6-4937-8d2a-52fe7490cafc/volumes"
Jul 23 22:52:26.259605 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:52:26.259571    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:52:26.261794 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:52:26.261772    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:53:34.656035 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.655993    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"]
Jul 23 22:53:34.656517 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.656326    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976e0f61-a6fc-42f5-b96c-25321b4ec9c5" containerName="payload-pre-processing"
Jul 23 22:53:34.656517 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.656337    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="976e0f61-a6fc-42f5-b96c-25321b4ec9c5" containerName="payload-pre-processing"
Jul 23 22:53:34.656517 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.656352    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3762f63-79a6-4937-8d2a-52fe7490cafc" containerName="payload-processing"
Jul 23 22:53:34.656517 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.656357    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3762f63-79a6-4937-8d2a-52fe7490cafc" containerName="payload-processing"
Jul 23 22:53:34.656517 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.656369    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b032ee-c6d9-4f02-9fea-fe46ea236bf6" containerName="maas-api"
Jul 23 22:53:34.656517 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.656375    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b032ee-c6d9-4f02-9fea-fe46ea236bf6" containerName="maas-api"
Jul 23 22:53:34.656517 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.656423    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="976e0f61-a6fc-42f5-b96c-25321b4ec9c5" containerName="payload-pre-processing"
Jul 23 22:53:34.656517 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.656434    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3762f63-79a6-4937-8d2a-52fe7490cafc" containerName="payload-processing"
Jul 23 22:53:34.656517 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.656440    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b032ee-c6d9-4f02-9fea-fe46ea236bf6" containerName="maas-api"
Jul 23 22:53:34.659442 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.659420    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:34.662265 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.662233    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 22:53:34.663384 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.663363    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 22:53:34.663488 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.663382    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d60afe1f-serving-cert\""
Jul 23 22:53:34.663488 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.663382    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vw8vx\""
Jul 23 22:53:34.669116 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.669091    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"]
Jul 23 22:53:34.785448 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.785408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7837a26-b45d-4979-8cb3-52ef64ed7127-maas-api-tls\") pod \"maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:34.785631 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.785540    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2swd\" (UniqueName: \"kubernetes.io/projected/f7837a26-b45d-4979-8cb3-52ef64ed7127-kube-api-access-d2swd\") pod \"maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:34.785631 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.785593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7837a26-b45d-4979-8cb3-52ef64ed7127-openshift-service-ca\") pod \"maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:34.886639 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.886594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2swd\" (UniqueName: \"kubernetes.io/projected/f7837a26-b45d-4979-8cb3-52ef64ed7127-kube-api-access-d2swd\") pod \"maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:34.886797 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.886653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7837a26-b45d-4979-8cb3-52ef64ed7127-openshift-service-ca\") pod \"maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:34.886797 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.886701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7837a26-b45d-4979-8cb3-52ef64ed7127-maas-api-tls\") pod \"maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:34.886875 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:53:34.886854    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-serving-cert: secret "maas-api-e2e-mt-d60afe1f-serving-cert" not found
Jul 23 22:53:34.886970 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:53:34.886959    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7837a26-b45d-4979-8cb3-52ef64ed7127-maas-api-tls podName:f7837a26-b45d-4979-8cb3-52ef64ed7127 nodeName:}" failed. No retries permitted until 2026-07-23 22:53:35.386909908 +0000 UTC m=+1869.704438705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7837a26-b45d-4979-8cb3-52ef64ed7127-maas-api-tls") pod "maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8" (UID: "f7837a26-b45d-4979-8cb3-52ef64ed7127") : secret "maas-api-e2e-mt-d60afe1f-serving-cert" not found
Jul 23 22:53:34.887400 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.887381    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7837a26-b45d-4979-8cb3-52ef64ed7127-openshift-service-ca\") pod \"maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:34.896784 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:34.896761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2swd\" (UniqueName: \"kubernetes.io/projected/f7837a26-b45d-4979-8cb3-52ef64ed7127-kube-api-access-d2swd\") pod \"maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:35.183589 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.183550    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"]
Jul 23 22:53:35.187180 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.187154    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:35.189682 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.189658    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d60afe1f-dockercfg-chckb\""
Jul 23 22:53:35.189798 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.189659    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d60afe1f\""
Jul 23 22:53:35.195585 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.195562    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"]
Jul 23 22:53:35.197692 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.197672    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"]
Jul 23 22:53:35.201265 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.201249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:35.211057 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.211035    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"]
Jul 23 22:53:35.290993 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.290964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m\" (UID: \"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:35.291168 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.291063    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxlf\" (UniqueName: \"kubernetes.io/projected/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-kube-api-access-rdxlf\") pod \"payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m\" (UID: \"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:35.392366 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.392326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxlf\" (UniqueName: \"kubernetes.io/projected/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-kube-api-access-rdxlf\") pod \"payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m\" (UID: \"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:35.392592 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.392377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d60afe1f-58bd478786-4plws\" (UID: \"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:35.392592 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.392409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv97n\" (UniqueName: \"kubernetes.io/projected/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-kube-api-access-tv97n\") pod \"payload-processing-e2e-mt-d60afe1f-58bd478786-4plws\" (UID: \"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:35.392592 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.392441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m\" (UID: \"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:35.392592 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.392530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7837a26-b45d-4979-8cb3-52ef64ed7127-maas-api-tls\") pod \"maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:35.393248 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.393224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m\" (UID: \"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:35.394894 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.394873    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7837a26-b45d-4979-8cb3-52ef64ed7127-maas-api-tls\") pod \"maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:35.399954 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.399912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxlf\" (UniqueName: \"kubernetes.io/projected/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-kube-api-access-rdxlf\") pod \"payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m\" (UID: \"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:35.493063 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.492979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv97n\" (UniqueName: \"kubernetes.io/projected/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-kube-api-access-tv97n\") pod \"payload-processing-e2e-mt-d60afe1f-58bd478786-4plws\" (UID: \"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:35.493203 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.493087    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d60afe1f-58bd478786-4plws\" (UID: \"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:35.493715 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.493697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d60afe1f-58bd478786-4plws\" (UID: \"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:35.499033 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.499005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:35.500778 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.500759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv97n\" (UniqueName: \"kubernetes.io/projected/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-kube-api-access-tv97n\") pod \"payload-processing-e2e-mt-d60afe1f-58bd478786-4plws\" (UID: \"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:35.512347 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.512314    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:35.571993 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.571964    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:35.656970 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.654425    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"]
Jul 23 22:53:35.659132 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:53:35.658897    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1401e05b_8ee8_4b4b_8cd4_bc634b5f67ca.slice/crio-8b3137405615c830f53bbb9c55e4cf341a4b5b0f15b73f7e758405c87c04f4c4 WatchSource:0}: Error finding container 8b3137405615c830f53bbb9c55e4cf341a4b5b0f15b73f7e758405c87c04f4c4: Status 404 returned error can't find the container with id 8b3137405615c830f53bbb9c55e4cf341a4b5b0f15b73f7e758405c87c04f4c4
Jul 23 22:53:35.735233 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.735142    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"]
Jul 23 22:53:35.736480 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:53:35.736451    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7837a26_b45d_4979_8cb3_52ef64ed7127.slice/crio-c826555eb8ea38edc2ad331b8d0cdb46563e0f7f78df45d95316e08b0dd6a90e WatchSource:0}: Error finding container c826555eb8ea38edc2ad331b8d0cdb46563e0f7f78df45d95316e08b0dd6a90e: Status 404 returned error can't find the container with id c826555eb8ea38edc2ad331b8d0cdb46563e0f7f78df45d95316e08b0dd6a90e
Jul 23 22:53:35.884283 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:35.884250    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"]
Jul 23 22:53:35.887128 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:53:35.887102    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c0fb8d_8ef3_472a_86fd_728b5cf43a3b.slice/crio-fe99d3dc68a5228d5cb3db12b85aaa6d13552fefbe5d7de1f97ff17c7705f929 WatchSource:0}: Error finding container fe99d3dc68a5228d5cb3db12b85aaa6d13552fefbe5d7de1f97ff17c7705f929: Status 404 returned error can't find the container with id fe99d3dc68a5228d5cb3db12b85aaa6d13552fefbe5d7de1f97ff17c7705f929
Jul 23 22:53:36.641276 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.641238    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8" event={"ID":"f7837a26-b45d-4979-8cb3-52ef64ed7127","Type":"ContainerStarted","Data":"14f181e71194e9e7218e8bbf79a0f78bc8b720c9e65ea47d401f432776283cab"}
Jul 23 22:53:36.641491 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.641300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8" event={"ID":"f7837a26-b45d-4979-8cb3-52ef64ed7127","Type":"ContainerStarted","Data":"c826555eb8ea38edc2ad331b8d0cdb46563e0f7f78df45d95316e08b0dd6a90e"}
Jul 23 22:53:36.641491 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.641397    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:36.645950 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.643546    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws" event={"ID":"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b","Type":"ContainerStarted","Data":"33ab0c3520b74e9691df2960a6112df1c2c532282f2178a201c16b01d88fe2e8"}
Jul 23 22:53:36.645950 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.643651    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:36.645950 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.643685    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws" event={"ID":"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b","Type":"ContainerStarted","Data":"fe99d3dc68a5228d5cb3db12b85aaa6d13552fefbe5d7de1f97ff17c7705f929"}
Jul 23 22:53:36.659025 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.658989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m" event={"ID":"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca","Type":"ContainerStarted","Data":"869e11d1c1faac2c1f894ce0c9bfbd5af420ec284af2cf61b63fe7b3819bc99f"}
Jul 23 22:53:36.659480 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.659043    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m" event={"ID":"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca","Type":"ContainerStarted","Data":"8b3137405615c830f53bbb9c55e4cf341a4b5b0f15b73f7e758405c87c04f4c4"}
Jul 23 22:53:36.659480 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.659238    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:36.660553 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.660505    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8" podStartSLOduration=2.191448359 podStartE2EDuration="2.660488791s" totalImagesPullingTime="469.040432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:53:36.658696385 +0000 UTC m=+1870.976225191" watchObservedRunningTime="2026-07-23 22:53:36.660488791 +0000 UTC m=+1870.978017601"
Jul 23 22:53:36.679792 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.679748    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws" podStartSLOduration=1.6797351200000001 podStartE2EDuration="1.67973512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:53:36.678112443 +0000 UTC m=+1870.995641251" watchObservedRunningTime="2026-07-23 22:53:36.67973512 +0000 UTC m=+1870.997263929"
Jul 23 22:53:36.694753 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:36.694710    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m" podStartSLOduration=1.694698114 podStartE2EDuration="1.694698114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:53:36.692903568 +0000 UTC m=+1871.010432374" watchObservedRunningTime="2026-07-23 22:53:36.694698114 +0000 UTC m=+1871.012226957"
Jul 23 22:53:42.669268 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:42.669238    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:47.665409 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:47.665375    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:47.665881 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:47.665453    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:56.548060 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.547977    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"]
Jul 23 22:53:56.548628 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.548191    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8" podUID="f7837a26-b45d-4979-8cb3-52ef64ed7127" containerName="maas-api" containerID="cri-o://14f181e71194e9e7218e8bbf79a0f78bc8b720c9e65ea47d401f432776283cab" gracePeriod=30
Jul 23 22:53:56.606285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.606248    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"]
Jul 23 22:53:56.606553 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.606523    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws" podUID="f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b" containerName="payload-processing" containerID="cri-o://33ab0c3520b74e9691df2960a6112df1c2c532282f2178a201c16b01d88fe2e8" gracePeriod=30
Jul 23 22:53:56.616907 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.616879    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"]
Jul 23 22:53:56.617338 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.617307    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m" podUID="1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca" containerName="payload-pre-processing" containerID="cri-o://869e11d1c1faac2c1f894ce0c9bfbd5af420ec284af2cf61b63fe7b3819bc99f" gracePeriod=30
Jul 23 22:53:56.765885 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.765060    2583 generic.go:356] "Generic (PLEG): container finished" podID="1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca" containerID="869e11d1c1faac2c1f894ce0c9bfbd5af420ec284af2cf61b63fe7b3819bc99f" exitCode=0
Jul 23 22:53:56.765885 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.765201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m" event={"ID":"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca","Type":"ContainerDied","Data":"869e11d1c1faac2c1f894ce0c9bfbd5af420ec284af2cf61b63fe7b3819bc99f"}
Jul 23 22:53:56.767191 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.767158    2583 generic.go:356] "Generic (PLEG): container finished" podID="f7837a26-b45d-4979-8cb3-52ef64ed7127" containerID="14f181e71194e9e7218e8bbf79a0f78bc8b720c9e65ea47d401f432776283cab" exitCode=0
Jul 23 22:53:56.767371 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.767193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8" event={"ID":"f7837a26-b45d-4979-8cb3-52ef64ed7127","Type":"ContainerDied","Data":"14f181e71194e9e7218e8bbf79a0f78bc8b720c9e65ea47d401f432776283cab"}
Jul 23 22:53:56.768956 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.768905    2583 generic.go:356] "Generic (PLEG): container finished" podID="f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b" containerID="33ab0c3520b74e9691df2960a6112df1c2c532282f2178a201c16b01d88fe2e8" exitCode=0
Jul 23 22:53:56.769085 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.768970    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws" event={"ID":"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b","Type":"ContainerDied","Data":"33ab0c3520b74e9691df2960a6112df1c2c532282f2178a201c16b01d88fe2e8"}
Jul 23 22:53:56.889021 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.888991    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:56.891810 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.891792    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:56.914326 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.914305    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:56.970251 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.970224    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7837a26-b45d-4979-8cb3-52ef64ed7127-maas-api-tls\") pod \"f7837a26-b45d-4979-8cb3-52ef64ed7127\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") "
Jul 23 22:53:56.970392 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.970256    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv97n\" (UniqueName: \"kubernetes.io/projected/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-kube-api-access-tv97n\") pod \"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b\" (UID: \"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b\") "
Jul 23 22:53:56.970392 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.970280    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7837a26-b45d-4979-8cb3-52ef64ed7127-openshift-service-ca\") pod \"f7837a26-b45d-4979-8cb3-52ef64ed7127\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") "
Jul 23 22:53:56.970502 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.970481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdxlf\" (UniqueName: \"kubernetes.io/projected/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-kube-api-access-rdxlf\") pod \"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca\" (UID: \"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca\") "
Jul 23 22:53:56.970591 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.970574    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2swd\" (UniqueName: \"kubernetes.io/projected/f7837a26-b45d-4979-8cb3-52ef64ed7127-kube-api-access-d2swd\") pod \"f7837a26-b45d-4979-8cb3-52ef64ed7127\" (UID: \"f7837a26-b45d-4979-8cb3-52ef64ed7127\") "
Jul 23 22:53:56.970653 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.970612    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-plugins-config-volume\") pod \"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca\" (UID: \"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca\") "
Jul 23 22:53:56.970653 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.970624    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7837a26-b45d-4979-8cb3-52ef64ed7127-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7837a26-b45d-4979-8cb3-52ef64ed7127" (UID: "f7837a26-b45d-4979-8cb3-52ef64ed7127"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:53:56.970653 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.970641    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-plugins-config-volume\") pod \"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b\" (UID: \"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b\") "
Jul 23 22:53:56.970938 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.970906    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7837a26-b45d-4979-8cb3-52ef64ed7127-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:53:56.971082 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.970956    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca" (UID: "1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:53:56.971271 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.971249    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b" (UID: "f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:53:56.972509 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.972486    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-kube-api-access-rdxlf" (OuterVolumeSpecName: "kube-api-access-rdxlf") pod "1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca" (UID: "1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca"). InnerVolumeSpecName "kube-api-access-rdxlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:53:56.972509 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.972491    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7837a26-b45d-4979-8cb3-52ef64ed7127-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7837a26-b45d-4979-8cb3-52ef64ed7127" (UID: "f7837a26-b45d-4979-8cb3-52ef64ed7127"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:53:56.972962 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.972939    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-kube-api-access-tv97n" (OuterVolumeSpecName: "kube-api-access-tv97n") pod "f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b" (UID: "f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b"). InnerVolumeSpecName "kube-api-access-tv97n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:53:56.973028 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:56.972917    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7837a26-b45d-4979-8cb3-52ef64ed7127-kube-api-access-d2swd" (OuterVolumeSpecName: "kube-api-access-d2swd") pod "f7837a26-b45d-4979-8cb3-52ef64ed7127" (UID: "f7837a26-b45d-4979-8cb3-52ef64ed7127"). InnerVolumeSpecName "kube-api-access-d2swd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:53:57.071465 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.071440    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7837a26-b45d-4979-8cb3-52ef64ed7127-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:53:57.071465 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.071465    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv97n\" (UniqueName: \"kubernetes.io/projected/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-kube-api-access-tv97n\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:53:57.071620 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.071475    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdxlf\" (UniqueName: \"kubernetes.io/projected/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-kube-api-access-rdxlf\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:53:57.071620 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.071484    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2swd\" (UniqueName: \"kubernetes.io/projected/f7837a26-b45d-4979-8cb3-52ef64ed7127-kube-api-access-d2swd\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:53:57.071620 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.071492    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:53:57.071620 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.071513    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:53:57.774693 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.774662    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"
Jul 23 22:53:57.774693 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.774670    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8" event={"ID":"f7837a26-b45d-4979-8cb3-52ef64ed7127","Type":"ContainerDied","Data":"c826555eb8ea38edc2ad331b8d0cdb46563e0f7f78df45d95316e08b0dd6a90e"}
Jul 23 22:53:57.775218 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.774738    2583 scope.go:117] "RemoveContainer" containerID="14f181e71194e9e7218e8bbf79a0f78bc8b720c9e65ea47d401f432776283cab"
Jul 23 22:53:57.776147 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.776130    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"
Jul 23 22:53:57.776219 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.776171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws" event={"ID":"f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b","Type":"ContainerDied","Data":"fe99d3dc68a5228d5cb3db12b85aaa6d13552fefbe5d7de1f97ff17c7705f929"}
Jul 23 22:53:57.777423 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.777404    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m" event={"ID":"1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca","Type":"ContainerDied","Data":"8b3137405615c830f53bbb9c55e4cf341a4b5b0f15b73f7e758405c87c04f4c4"}
Jul 23 22:53:57.777566 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.777426    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"
Jul 23 22:53:57.784217 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.784196    2583 scope.go:117] "RemoveContainer" containerID="33ab0c3520b74e9691df2960a6112df1c2c532282f2178a201c16b01d88fe2e8"
Jul 23 22:53:57.793207 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.793190    2583 scope.go:117] "RemoveContainer" containerID="869e11d1c1faac2c1f894ce0c9bfbd5af420ec284af2cf61b63fe7b3819bc99f"
Jul 23 22:53:57.804553 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.802795    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"]
Jul 23 22:53:57.806680 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.806658    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d60afe1f-777545ff76-b7b8m"]
Jul 23 22:53:57.819804 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.819753    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"]
Jul 23 22:53:57.823095 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.823075    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d60afe1f-85f74bd6fc-tgql8"]
Jul 23 22:53:57.832966 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.832946    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"]
Jul 23 22:53:57.835166 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:57.835147    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d60afe1f-58bd478786-4plws"]
Jul 23 22:53:58.228911 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:58.228835    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca" path="/var/lib/kubelet/pods/1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca/volumes"
Jul 23 22:53:58.229264 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:58.229248    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b" path="/var/lib/kubelet/pods/f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b/volumes"
Jul 23 22:53:58.229656 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:53:58.229639    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7837a26-b45d-4979-8cb3-52ef64ed7127" path="/var/lib/kubelet/pods/f7837a26-b45d-4979-8cb3-52ef64ed7127/volumes"
Jul 23 22:54:31.716775 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.716731    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"]
Jul 23 22:54:31.717336 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.717167    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca" containerName="payload-pre-processing"
Jul 23 22:54:31.717336 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.717186    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca" containerName="payload-pre-processing"
Jul 23 22:54:31.717336 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.717215    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b" containerName="payload-processing"
Jul 23 22:54:31.717336 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.717225    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b" containerName="payload-processing"
Jul 23 22:54:31.717336 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.717238    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7837a26-b45d-4979-8cb3-52ef64ed7127" containerName="maas-api"
Jul 23 22:54:31.717336 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.717246    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7837a26-b45d-4979-8cb3-52ef64ed7127" containerName="maas-api"
Jul 23 22:54:31.717640 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.717352    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c0fb8d-8ef3-472a-86fd-728b5cf43a3b" containerName="payload-processing"
Jul 23 22:54:31.717640 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.717364    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7837a26-b45d-4979-8cb3-52ef64ed7127" containerName="maas-api"
Jul 23 22:54:31.717640 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.717376    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1401e05b-8ee8-4b4b-8cd4-bc634b5f67ca" containerName="payload-pre-processing"
Jul 23 22:54:31.720479 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.720458    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:31.723275 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.723254    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vw8vx\""
Jul 23 22:54:31.723440 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.723283    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b2ff72b5-serving-cert\""
Jul 23 22:54:31.724481 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.724459    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 22:54:31.724658 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.724640    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 22:54:31.736166 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.736132    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"]
Jul 23 22:54:31.765071 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.765040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9395a083-5297-43e6-909e-5235d969ecc8-openshift-service-ca\") pod \"maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:31.765210 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.765080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls\") pod \"maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:31.765210 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.765172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqnph\" (UniqueName: \"kubernetes.io/projected/9395a083-5297-43e6-909e-5235d969ecc8-kube-api-access-vqnph\") pod \"maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:31.865673 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.865638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9395a083-5297-43e6-909e-5235d969ecc8-openshift-service-ca\") pod \"maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:31.865878 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.865694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls\") pod \"maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:31.865878 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.865763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqnph\" (UniqueName: \"kubernetes.io/projected/9395a083-5297-43e6-909e-5235d969ecc8-kube-api-access-vqnph\") pod \"maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:31.866032 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:54:31.865875    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-serving-cert: secret "maas-api-e2e-mt-b2ff72b5-serving-cert" not found
Jul 23 22:54:31.866032 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:54:31.865994    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls podName:9395a083-5297-43e6-909e-5235d969ecc8 nodeName:}" failed. No retries permitted until 2026-07-23 22:54:32.365973131 +0000 UTC m=+1926.683501921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls") pod "maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk" (UID: "9395a083-5297-43e6-909e-5235d969ecc8") : secret "maas-api-e2e-mt-b2ff72b5-serving-cert" not found
Jul 23 22:54:31.866401 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.866379    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9395a083-5297-43e6-909e-5235d969ecc8-openshift-service-ca\") pod \"maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:31.875053 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:31.875026    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqnph\" (UniqueName: \"kubernetes.io/projected/9395a083-5297-43e6-909e-5235d969ecc8-kube-api-access-vqnph\") pod \"maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:32.369831 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.369790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls\") pod \"maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:32.370014 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:54:32.369946    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-serving-cert: secret "maas-api-e2e-mt-b2ff72b5-serving-cert" not found
Jul 23 22:54:32.370065 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:54:32.370014    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls podName:9395a083-5297-43e6-909e-5235d969ecc8 nodeName:}" failed. No retries permitted until 2026-07-23 22:54:33.369997273 +0000 UTC m=+1927.687526063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls") pod "maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk" (UID: "9395a083-5297-43e6-909e-5235d969ecc8") : secret "maas-api-e2e-mt-b2ff72b5-serving-cert" not found
Jul 23 22:54:32.427597 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.427564    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"]
Jul 23 22:54:32.431126 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.431108    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:54:32.433708 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.433687    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b2ff72b5\""
Jul 23 22:54:32.433845 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.433724    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b2ff72b5-dockercfg-tqrml\""
Jul 23 22:54:32.440050 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.440000    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"]
Jul 23 22:54:32.442394 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.442369    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"]
Jul 23 22:54:32.446131 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.446113    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:54:32.454744 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.454712    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"]
Jul 23 22:54:32.470312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.470287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ea74cf-df6a-4659-86f4-ff6a000d9698-plugins-config-volume\") pod \"payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh\" (UID: \"96ea74cf-df6a-4659-86f4-ff6a000d9698\") " pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:54:32.470421 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.470355    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e549c11-9e61-4b97-8672-bb621b8b6f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt\" (UID: \"4e549c11-9e61-4b97-8672-bb621b8b6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:54:32.470421 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.470384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzghr\" (UniqueName: \"kubernetes.io/projected/96ea74cf-df6a-4659-86f4-ff6a000d9698-kube-api-access-gzghr\") pod \"payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh\" (UID: \"96ea74cf-df6a-4659-86f4-ff6a000d9698\") " pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:54:32.470524 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.470422    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s67xn\" (UniqueName: \"kubernetes.io/projected/4e549c11-9e61-4b97-8672-bb621b8b6f14-kube-api-access-s67xn\") pod \"payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt\" (UID: \"4e549c11-9e61-4b97-8672-bb621b8b6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:54:32.571315 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.571277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ea74cf-df6a-4659-86f4-ff6a000d9698-plugins-config-volume\") pod \"payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh\" (UID: \"96ea74cf-df6a-4659-86f4-ff6a000d9698\") " pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:54:32.571485 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.571344    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e549c11-9e61-4b97-8672-bb621b8b6f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt\" (UID: \"4e549c11-9e61-4b97-8672-bb621b8b6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:54:32.571485 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.571370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzghr\" (UniqueName: \"kubernetes.io/projected/96ea74cf-df6a-4659-86f4-ff6a000d9698-kube-api-access-gzghr\") pod \"payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh\" (UID: \"96ea74cf-df6a-4659-86f4-ff6a000d9698\") " pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:54:32.571485 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.571397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67xn\" (UniqueName: \"kubernetes.io/projected/4e549c11-9e61-4b97-8672-bb621b8b6f14-kube-api-access-s67xn\") pod \"payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt\" (UID: \"4e549c11-9e61-4b97-8672-bb621b8b6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:54:32.572132 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.572110    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ea74cf-df6a-4659-86f4-ff6a000d9698-plugins-config-volume\") pod \"payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh\" (UID: \"96ea74cf-df6a-4659-86f4-ff6a000d9698\") " pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:54:32.572327 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.572304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e549c11-9e61-4b97-8672-bb621b8b6f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt\" (UID: \"4e549c11-9e61-4b97-8672-bb621b8b6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:54:32.580560 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.580539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67xn\" (UniqueName: \"kubernetes.io/projected/4e549c11-9e61-4b97-8672-bb621b8b6f14-kube-api-access-s67xn\") pod \"payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt\" (UID: \"4e549c11-9e61-4b97-8672-bb621b8b6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:54:32.580560 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.580551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzghr\" (UniqueName: \"kubernetes.io/projected/96ea74cf-df6a-4659-86f4-ff6a000d9698-kube-api-access-gzghr\") pod \"payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh\" (UID: \"96ea74cf-df6a-4659-86f4-ff6a000d9698\") " pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:54:32.744346 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.744248    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:54:32.758573 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.758550    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:54:32.892830 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.892797    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"]
Jul 23 22:54:32.895744 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:54:32.895716    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e549c11_9e61_4b97_8672_bb621b8b6f14.slice/crio-5778e2b49c0a36bd090dc51751b25c6427178432b4f52e320e7e8a74086ac4f1 WatchSource:0}: Error finding container 5778e2b49c0a36bd090dc51751b25c6427178432b4f52e320e7e8a74086ac4f1: Status 404 returned error can't find the container with id 5778e2b49c0a36bd090dc51751b25c6427178432b4f52e320e7e8a74086ac4f1
Jul 23 22:54:32.923613 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.923591    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"]
Jul 23 22:54:32.931014 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:54:32.930976    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ea74cf_df6a_4659_86f4_ff6a000d9698.slice/crio-5b4dbddc6b58cb3ee31a4efe221c81277ff666836cf5c71b9ac0b7240766b6f4 WatchSource:0}: Error finding container 5b4dbddc6b58cb3ee31a4efe221c81277ff666836cf5c71b9ac0b7240766b6f4: Status 404 returned error can't find the container with id 5b4dbddc6b58cb3ee31a4efe221c81277ff666836cf5c71b9ac0b7240766b6f4
Jul 23 22:54:32.945936 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.945896    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh" event={"ID":"96ea74cf-df6a-4659-86f4-ff6a000d9698","Type":"ContainerStarted","Data":"5b4dbddc6b58cb3ee31a4efe221c81277ff666836cf5c71b9ac0b7240766b6f4"}
Jul 23 22:54:32.947137 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:32.947113    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt" event={"ID":"4e549c11-9e61-4b97-8672-bb621b8b6f14","Type":"ContainerStarted","Data":"5778e2b49c0a36bd090dc51751b25c6427178432b4f52e320e7e8a74086ac4f1"}
Jul 23 22:54:33.379892 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:33.379846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls\") pod \"maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:33.382290 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:33.382268    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls\") pod \"maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:33.534081 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:33.534046    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:33.667632 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:33.667606    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"]
Jul 23 22:54:33.669508 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:54:33.669472    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9395a083_5297_43e6_909e_5235d969ecc8.slice/crio-909103137f66ab31dba6f97769958f381011d810be8720632d3a822a90c44c55 WatchSource:0}: Error finding container 909103137f66ab31dba6f97769958f381011d810be8720632d3a822a90c44c55: Status 404 returned error can't find the container with id 909103137f66ab31dba6f97769958f381011d810be8720632d3a822a90c44c55
Jul 23 22:54:33.954767 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:33.954677    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh" event={"ID":"96ea74cf-df6a-4659-86f4-ff6a000d9698","Type":"ContainerStarted","Data":"5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02"}
Jul 23 22:54:33.955242 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:33.954781    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:54:33.956451 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:33.956427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt" event={"ID":"4e549c11-9e61-4b97-8672-bb621b8b6f14","Type":"ContainerStarted","Data":"87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d"}
Jul 23 22:54:33.956573 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:33.956468    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:54:33.957509 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:33.957491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk" event={"ID":"9395a083-5297-43e6-909e-5235d969ecc8","Type":"ContainerStarted","Data":"909103137f66ab31dba6f97769958f381011d810be8720632d3a822a90c44c55"}
Jul 23 22:54:33.974798 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:33.974749    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh" podStartSLOduration=1.9747302869999999 podStartE2EDuration="1.974730287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:54:33.973260595 +0000 UTC m=+1928.290789404" watchObservedRunningTime="2026-07-23 22:54:33.974730287 +0000 UTC m=+1928.292259095"
Jul 23 22:54:33.990216 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:33.990180    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt" podStartSLOduration=1.990167108 podStartE2EDuration="1.990167108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:54:33.989152156 +0000 UTC m=+1928.306680965" watchObservedRunningTime="2026-07-23 22:54:33.990167108 +0000 UTC m=+1928.307695914"
Jul 23 22:54:34.963713 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:34.963671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk" event={"ID":"9395a083-5297-43e6-909e-5235d969ecc8","Type":"ContainerStarted","Data":"ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0"}
Jul 23 22:54:34.982272 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:34.982225    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk" podStartSLOduration=3.685579768 podStartE2EDuration="3.982209947s" totalImagesPullingTime="296.630179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:54:34.980797482 +0000 UTC m=+1929.298326291" watchObservedRunningTime="2026-07-23 22:54:34.982209947 +0000 UTC m=+1929.299738755"
Jul 23 22:54:35.969142 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:35.969107    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:41.979743 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:41.979711    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:54:44.966128 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:44.966095    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:54:44.966564 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:54:44.966175    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:55:15.905744 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:15.905711    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"]
Jul 23 22:55:15.909390 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:15.909366    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"
Jul 23 22:55:15.911817 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:15.911794    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6cq4\""
Jul 23 22:55:15.916855 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:15.916824    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"]
Jul 23 22:55:16.039829 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:16.039798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c932c285-46ac-4397-8047-63c0e5b68edb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9\" (UID: \"c932c285-46ac-4397-8047-63c0e5b68edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"
Jul 23 22:55:16.141103 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:16.141069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c932c285-46ac-4397-8047-63c0e5b68edb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9\" (UID: \"c932c285-46ac-4397-8047-63c0e5b68edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"
Jul 23 22:55:16.141690 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:16.141668    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c932c285-46ac-4397-8047-63c0e5b68edb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9\" (UID: \"c932c285-46ac-4397-8047-63c0e5b68edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"
Jul 23 22:55:16.220992 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:16.220887    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"
Jul 23 22:55:16.357216 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:16.357180    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"]
Jul 23 22:55:16.360126 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:55:16.360099    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc932c285_46ac_4397_8047_63c0e5b68edb.slice/crio-d23e8ed65f5e349ce00a2b2f58520d6dad2a90da35ec51a785e6366acb553cb0 WatchSource:0}: Error finding container d23e8ed65f5e349ce00a2b2f58520d6dad2a90da35ec51a785e6366acb553cb0: Status 404 returned error can't find the container with id d23e8ed65f5e349ce00a2b2f58520d6dad2a90da35ec51a785e6366acb553cb0
Jul 23 22:55:17.161625 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:17.161593    2583 generic.go:356] "Generic (PLEG): container finished" podID="c932c285-46ac-4397-8047-63c0e5b68edb" containerID="a2c16972da19aa697ff20df064a3280b8e3bbc5cf319af5bd3b94a10757184f0" exitCode=0
Jul 23 22:55:17.162023 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:17.161688    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9" event={"ID":"c932c285-46ac-4397-8047-63c0e5b68edb","Type":"ContainerDied","Data":"a2c16972da19aa697ff20df064a3280b8e3bbc5cf319af5bd3b94a10757184f0"}
Jul 23 22:55:17.162023 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:17.161728    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9" event={"ID":"c932c285-46ac-4397-8047-63c0e5b68edb","Type":"ContainerStarted","Data":"d23e8ed65f5e349ce00a2b2f58520d6dad2a90da35ec51a785e6366acb553cb0"}
Jul 23 22:55:18.295786 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:18.295759    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"
Jul 23 22:55:18.463383 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:18.463275    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c932c285-46ac-4397-8047-63c0e5b68edb-maas-api-service-ca\") pod \"c932c285-46ac-4397-8047-63c0e5b68edb\" (UID: \"c932c285-46ac-4397-8047-63c0e5b68edb\") "
Jul 23 22:55:18.463629 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:18.463605    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c932c285-46ac-4397-8047-63c0e5b68edb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c932c285-46ac-4397-8047-63c0e5b68edb" (UID: "c932c285-46ac-4397-8047-63c0e5b68edb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:55:18.564274 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:18.564242    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c932c285-46ac-4397-8047-63c0e5b68edb-maas-api-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:55:19.173621 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:19.173594    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"
Jul 23 22:55:19.173785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:19.173629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9" event={"ID":"c932c285-46ac-4397-8047-63c0e5b68edb","Type":"ContainerDied","Data":"d23e8ed65f5e349ce00a2b2f58520d6dad2a90da35ec51a785e6366acb553cb0"}
Jul 23 22:55:19.173785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:19.173673    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d23e8ed65f5e349ce00a2b2f58520d6dad2a90da35ec51a785e6366acb553cb0"
Jul 23 22:55:26.373182 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:26.373151    2583 scope.go:117] "RemoveContainer" containerID="86a6be115efe6b661dc4d8972cf4a7358dd83d121fbe9274dbaf4d3bc703f734"
Jul 23 22:55:31.289411 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.289376    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"]
Jul 23 22:55:31.289879 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.289642    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk" podUID="9395a083-5297-43e6-909e-5235d969ecc8" containerName="maas-api" containerID="cri-o://ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0" gracePeriod=30
Jul 23 22:55:31.343793 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.343758    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"]
Jul 23 22:55:31.344059 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.343968    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh" podUID="96ea74cf-df6a-4659-86f4-ff6a000d9698" containerName="payload-processing" containerID="cri-o://5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02" gracePeriod=30
Jul 23 22:55:31.350338 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.350311    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"]
Jul 23 22:55:31.350550 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.350524    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt" podUID="4e549c11-9e61-4b97-8672-bb621b8b6f14" containerName="payload-pre-processing" containerID="cri-o://87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d" gracePeriod=30
Jul 23 22:55:31.630406 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.630374    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:55:31.655322 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.655296    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:55:31.661172 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.661148    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:55:31.679697 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.679669    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ea74cf-df6a-4659-86f4-ff6a000d9698-plugins-config-volume\") pod \"96ea74cf-df6a-4659-86f4-ff6a000d9698\" (UID: \"96ea74cf-df6a-4659-86f4-ff6a000d9698\") "
Jul 23 22:55:31.679803 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.679729    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e549c11-9e61-4b97-8672-bb621b8b6f14-plugins-config-volume\") pod \"4e549c11-9e61-4b97-8672-bb621b8b6f14\" (UID: \"4e549c11-9e61-4b97-8672-bb621b8b6f14\") "
Jul 23 22:55:31.679803 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.679772    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9395a083-5297-43e6-909e-5235d969ecc8-openshift-service-ca\") pod \"9395a083-5297-43e6-909e-5235d969ecc8\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") "
Jul 23 22:55:31.679803 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.679799    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls\") pod \"9395a083-5297-43e6-909e-5235d969ecc8\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") "
Jul 23 22:55:31.679946 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.679833    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzghr\" (UniqueName: \"kubernetes.io/projected/96ea74cf-df6a-4659-86f4-ff6a000d9698-kube-api-access-gzghr\") pod \"96ea74cf-df6a-4659-86f4-ff6a000d9698\" (UID: \"96ea74cf-df6a-4659-86f4-ff6a000d9698\") "
Jul 23 22:55:31.679946 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.679868    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s67xn\" (UniqueName: \"kubernetes.io/projected/4e549c11-9e61-4b97-8672-bb621b8b6f14-kube-api-access-s67xn\") pod \"4e549c11-9e61-4b97-8672-bb621b8b6f14\" (UID: \"4e549c11-9e61-4b97-8672-bb621b8b6f14\") "
Jul 23 22:55:31.679946 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.679900    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqnph\" (UniqueName: \"kubernetes.io/projected/9395a083-5297-43e6-909e-5235d969ecc8-kube-api-access-vqnph\") pod \"9395a083-5297-43e6-909e-5235d969ecc8\" (UID: \"9395a083-5297-43e6-909e-5235d969ecc8\") "
Jul 23 22:55:31.680142 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.680115    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ea74cf-df6a-4659-86f4-ff6a000d9698-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96ea74cf-df6a-4659-86f4-ff6a000d9698" (UID: "96ea74cf-df6a-4659-86f4-ff6a000d9698"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:55:31.680207 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.680166    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e549c11-9e61-4b97-8672-bb621b8b6f14-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e549c11-9e61-4b97-8672-bb621b8b6f14" (UID: "4e549c11-9e61-4b97-8672-bb621b8b6f14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:55:31.680280 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.680181    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9395a083-5297-43e6-909e-5235d969ecc8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9395a083-5297-43e6-909e-5235d969ecc8" (UID: "9395a083-5297-43e6-909e-5235d969ecc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:55:31.680280 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.680231    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ea74cf-df6a-4659-86f4-ff6a000d9698-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:55:31.682212 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.682180    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ea74cf-df6a-4659-86f4-ff6a000d9698-kube-api-access-gzghr" (OuterVolumeSpecName: "kube-api-access-gzghr") pod "96ea74cf-df6a-4659-86f4-ff6a000d9698" (UID: "96ea74cf-df6a-4659-86f4-ff6a000d9698"). InnerVolumeSpecName "kube-api-access-gzghr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:55:31.682318 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.682227    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e549c11-9e61-4b97-8672-bb621b8b6f14-kube-api-access-s67xn" (OuterVolumeSpecName: "kube-api-access-s67xn") pod "4e549c11-9e61-4b97-8672-bb621b8b6f14" (UID: "4e549c11-9e61-4b97-8672-bb621b8b6f14"). InnerVolumeSpecName "kube-api-access-s67xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:55:31.682386 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.682352    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9395a083-5297-43e6-909e-5235d969ecc8" (UID: "9395a083-5297-43e6-909e-5235d969ecc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:55:31.682453 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.682437    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9395a083-5297-43e6-909e-5235d969ecc8-kube-api-access-vqnph" (OuterVolumeSpecName: "kube-api-access-vqnph") pod "9395a083-5297-43e6-909e-5235d969ecc8" (UID: "9395a083-5297-43e6-909e-5235d969ecc8"). InnerVolumeSpecName "kube-api-access-vqnph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:55:31.780758 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.780720    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e549c11-9e61-4b97-8672-bb621b8b6f14-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:55:31.780758 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.780754    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9395a083-5297-43e6-909e-5235d969ecc8-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:55:31.780758 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.780764    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9395a083-5297-43e6-909e-5235d969ecc8-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:55:31.781033 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.780773    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzghr\" (UniqueName: \"kubernetes.io/projected/96ea74cf-df6a-4659-86f4-ff6a000d9698-kube-api-access-gzghr\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:55:31.781033 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.780783    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s67xn\" (UniqueName: \"kubernetes.io/projected/4e549c11-9e61-4b97-8672-bb621b8b6f14-kube-api-access-s67xn\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:55:31.781033 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:31.780792    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqnph\" (UniqueName: \"kubernetes.io/projected/9395a083-5297-43e6-909e-5235d969ecc8-kube-api-access-vqnph\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:55:32.234200 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.234150    2583 generic.go:356] "Generic (PLEG): container finished" podID="9395a083-5297-43e6-909e-5235d969ecc8" containerID="ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0" exitCode=0
Jul 23 22:55:32.234396 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.234218    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"
Jul 23 22:55:32.234396 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.234223    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk" event={"ID":"9395a083-5297-43e6-909e-5235d969ecc8","Type":"ContainerDied","Data":"ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0"}
Jul 23 22:55:32.234396 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.234305    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk" event={"ID":"9395a083-5297-43e6-909e-5235d969ecc8","Type":"ContainerDied","Data":"909103137f66ab31dba6f97769958f381011d810be8720632d3a822a90c44c55"}
Jul 23 22:55:32.234396 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.234337    2583 scope.go:117] "RemoveContainer" containerID="ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0"
Jul 23 22:55:32.235838 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.235802    2583 generic.go:356] "Generic (PLEG): container finished" podID="96ea74cf-df6a-4659-86f4-ff6a000d9698" containerID="5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02" exitCode=0
Jul 23 22:55:32.236050 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.235949    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"
Jul 23 22:55:32.236050 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.235968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh" event={"ID":"96ea74cf-df6a-4659-86f4-ff6a000d9698","Type":"ContainerDied","Data":"5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02"}
Jul 23 22:55:32.236050 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.236018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh" event={"ID":"96ea74cf-df6a-4659-86f4-ff6a000d9698","Type":"ContainerDied","Data":"5b4dbddc6b58cb3ee31a4efe221c81277ff666836cf5c71b9ac0b7240766b6f4"}
Jul 23 22:55:32.237434 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.237411    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e549c11-9e61-4b97-8672-bb621b8b6f14" containerID="87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d" exitCode=0
Jul 23 22:55:32.237563 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.237490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt" event={"ID":"4e549c11-9e61-4b97-8672-bb621b8b6f14","Type":"ContainerDied","Data":"87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d"}
Jul 23 22:55:32.237666 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.237592    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt" event={"ID":"4e549c11-9e61-4b97-8672-bb621b8b6f14","Type":"ContainerDied","Data":"5778e2b49c0a36bd090dc51751b25c6427178432b4f52e320e7e8a74086ac4f1"}
Jul 23 22:55:32.237666 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.237520    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"
Jul 23 22:55:32.245778 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.245753    2583 scope.go:117] "RemoveContainer" containerID="ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0"
Jul 23 22:55:32.246117 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:55:32.246092    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0\": container with ID starting with ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0 not found: ID does not exist" containerID="ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0"
Jul 23 22:55:32.246181 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.246126    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0"} err="failed to get container status \"ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0\": rpc error: code = NotFound desc = could not find container \"ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0\": container with ID starting with ee0568d04fcdf648ac0cb65487ce12f32dab36b89c431040a0269e0c7b49c2f0 not found: ID does not exist"
Jul 23 22:55:32.246181 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.246148    2583 scope.go:117] "RemoveContainer" containerID="5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02"
Jul 23 22:55:32.256523 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.256506    2583 scope.go:117] "RemoveContainer" containerID="5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02"
Jul 23 22:55:32.256830 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:55:32.256810    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02\": container with ID starting with 5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02 not found: ID does not exist" containerID="5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02"
Jul 23 22:55:32.256888 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.256836    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02"} err="failed to get container status \"5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02\": rpc error: code = NotFound desc = could not find container \"5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02\": container with ID starting with 5d6424f4c7ec8a9099ace62728ed869447e3058caab5d772303c51cf0d098e02 not found: ID does not exist"
Jul 23 22:55:32.256888 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.256851    2583 scope.go:117] "RemoveContainer" containerID="87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d"
Jul 23 22:55:32.264366 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.264348    2583 scope.go:117] "RemoveContainer" containerID="87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d"
Jul 23 22:55:32.264638 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:55:32.264606    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d\": container with ID starting with 87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d not found: ID does not exist" containerID="87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d"
Jul 23 22:55:32.264693 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.264639    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d"} err="failed to get container status \"87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d\": rpc error: code = NotFound desc = could not find container \"87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d\": container with ID starting with 87d2135f6da08c35993fcca4a08d697cd84882314d51ec9916eee78df058163d not found: ID does not exist"
Jul 23 22:55:32.264964 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.264949    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"]
Jul 23 22:55:32.268423 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.268401    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b2ff72b5-64f777f67d-khmkh"]
Jul 23 22:55:32.278399 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.278380    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"]
Jul 23 22:55:32.282275 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.282254    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b2ff72b5-c8fb875dd-q2rtt"]
Jul 23 22:55:32.292136 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.292092    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"]
Jul 23 22:55:32.294317 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:32.294298    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2ff72b5-59fc78f9fd-mr8dk"]
Jul 23 22:55:34.229347 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:34.229311    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e549c11-9e61-4b97-8672-bb621b8b6f14" path="/var/lib/kubelet/pods/4e549c11-9e61-4b97-8672-bb621b8b6f14/volumes"
Jul 23 22:55:34.229723 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:34.229697    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9395a083-5297-43e6-909e-5235d969ecc8" path="/var/lib/kubelet/pods/9395a083-5297-43e6-909e-5235d969ecc8/volumes"
Jul 23 22:55:34.230050 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:34.230034    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96ea74cf-df6a-4659-86f4-ff6a000d9698" path="/var/lib/kubelet/pods/96ea74cf-df6a-4659-86f4-ff6a000d9698/volumes"
Jul 23 22:55:35.981582 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:35.981550    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"]
Jul 23 22:55:35.984443 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:35.984415    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b2ff72b5-kf6n9"]
Jul 23 22:55:36.229285 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:55:36.229252    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c932c285-46ac-4397-8047-63c0e5b68edb" path="/var/lib/kubelet/pods/c932c285-46ac-4397-8047-63c0e5b68edb/volumes"
Jul 23 22:57:26.294768 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:26.294738    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:57:26.298209 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:26.298178    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 22:57:32.840685 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.840643    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"]
Jul 23 22:57:32.841283 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841134    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e549c11-9e61-4b97-8672-bb621b8b6f14" containerName="payload-pre-processing"
Jul 23 22:57:32.841283 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841155    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e549c11-9e61-4b97-8672-bb621b8b6f14" containerName="payload-pre-processing"
Jul 23 22:57:32.841283 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841175    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ea74cf-df6a-4659-86f4-ff6a000d9698" containerName="payload-processing"
Jul 23 22:57:32.841283 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841185    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ea74cf-df6a-4659-86f4-ff6a000d9698" containerName="payload-processing"
Jul 23 22:57:32.841283 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841198    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c932c285-46ac-4397-8047-63c0e5b68edb" containerName="revoke-keys"
Jul 23 22:57:32.841283 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841207    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c932c285-46ac-4397-8047-63c0e5b68edb" containerName="revoke-keys"
Jul 23 22:57:32.841283 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841227    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9395a083-5297-43e6-909e-5235d969ecc8" containerName="maas-api"
Jul 23 22:57:32.841283 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841236    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9395a083-5297-43e6-909e-5235d969ecc8" containerName="maas-api"
Jul 23 22:57:32.841694 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841320    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e549c11-9e61-4b97-8672-bb621b8b6f14" containerName="payload-pre-processing"
Jul 23 22:57:32.841694 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841333    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="96ea74cf-df6a-4659-86f4-ff6a000d9698" containerName="payload-processing"
Jul 23 22:57:32.841694 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841345    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9395a083-5297-43e6-909e-5235d969ecc8" containerName="maas-api"
Jul 23 22:57:32.841694 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.841355    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c932c285-46ac-4397-8047-63c0e5b68edb" containerName="revoke-keys"
Jul 23 22:57:32.844490 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.844467    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:32.847407 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.847385    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c70c7c-serving-cert\""
Jul 23 22:57:32.847532 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.847474    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 22:57:32.848507 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.848487    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 22:57:32.848620 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.848490    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vw8vx\""
Jul 23 22:57:32.862467 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.862428    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"]
Jul 23 22:57:32.891677 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.891638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bfcb41-50de-4730-8519-8ba50c677131-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:32.891881 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.891783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68bfcb41-50de-4730-8519-8ba50c677131-maas-api-tls\") pod \"maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:32.891881 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.891824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8xbf\" (UniqueName: \"kubernetes.io/projected/68bfcb41-50de-4730-8519-8ba50c677131-kube-api-access-v8xbf\") pod \"maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:32.992983 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.992944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68bfcb41-50de-4730-8519-8ba50c677131-maas-api-tls\") pod \"maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:32.992983 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.992985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xbf\" (UniqueName: \"kubernetes.io/projected/68bfcb41-50de-4730-8519-8ba50c677131-kube-api-access-v8xbf\") pod \"maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:32.993246 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.993029    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bfcb41-50de-4730-8519-8ba50c677131-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:32.993246 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:57:32.993029    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-serving-cert: secret "maas-api-e2e-inf-a-c70c7c-serving-cert" not found
Jul 23 22:57:32.993246 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:57:32.993164    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68bfcb41-50de-4730-8519-8ba50c677131-maas-api-tls podName:68bfcb41-50de-4730-8519-8ba50c677131 nodeName:}" failed. No retries permitted until 2026-07-23 22:57:33.493147405 +0000 UTC m=+2107.810676192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68bfcb41-50de-4730-8519-8ba50c677131-maas-api-tls") pod "maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r" (UID: "68bfcb41-50de-4730-8519-8ba50c677131") : secret "maas-api-e2e-inf-a-c70c7c-serving-cert" not found
Jul 23 22:57:32.993610 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:32.993588    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bfcb41-50de-4730-8519-8ba50c677131-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:33.002259 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.002233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xbf\" (UniqueName: \"kubernetes.io/projected/68bfcb41-50de-4730-8519-8ba50c677131-kube-api-access-v8xbf\") pod \"maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:33.498486 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.498454    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68bfcb41-50de-4730-8519-8ba50c677131-maas-api-tls\") pod \"maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:33.500850 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.500819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68bfcb41-50de-4730-8519-8ba50c677131-maas-api-tls\") pod \"maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:33.557987 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.557950    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"]
Jul 23 22:57:33.561418 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.561395    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:57:33.564129 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.564104    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c70c7c\""
Jul 23 22:57:33.565509 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.565488    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c70c7c-dockercfg-8flfr\""
Jul 23 22:57:33.574405 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.574380    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"]
Jul 23 22:57:33.598816 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.598790    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"]
Jul 23 22:57:33.599106 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.599084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcbb03f-1a82-476b-8db4-6f95117625a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g\" (UID: \"ddcbb03f-1a82-476b-8db4-6f95117625a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:57:33.599173 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.599128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpn5t\" (UniqueName: \"kubernetes.io/projected/ddcbb03f-1a82-476b-8db4-6f95117625a6-kube-api-access-wpn5t\") pod \"payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g\" (UID: \"ddcbb03f-1a82-476b-8db4-6f95117625a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:57:33.602006 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.601991    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:57:33.612846 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.612820    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"]
Jul 23 22:57:33.699743 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.699706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpn5t\" (UniqueName: \"kubernetes.io/projected/ddcbb03f-1a82-476b-8db4-6f95117625a6-kube-api-access-wpn5t\") pod \"payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g\" (UID: \"ddcbb03f-1a82-476b-8db4-6f95117625a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:57:33.699958 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.699865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nlk\" (UniqueName: \"kubernetes.io/projected/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-kube-api-access-s9nlk\") pod \"payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b\" (UID: \"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:57:33.699958 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.699896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcbb03f-1a82-476b-8db4-6f95117625a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g\" (UID: \"ddcbb03f-1a82-476b-8db4-6f95117625a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:57:33.699958 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.699917    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b\" (UID: \"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:57:33.700675 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.700643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcbb03f-1a82-476b-8db4-6f95117625a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g\" (UID: \"ddcbb03f-1a82-476b-8db4-6f95117625a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:57:33.708356 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.708331    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpn5t\" (UniqueName: \"kubernetes.io/projected/ddcbb03f-1a82-476b-8db4-6f95117625a6-kube-api-access-wpn5t\") pod \"payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g\" (UID: \"ddcbb03f-1a82-476b-8db4-6f95117625a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:57:33.754685 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.754597    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:33.800430 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.800400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nlk\" (UniqueName: \"kubernetes.io/projected/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-kube-api-access-s9nlk\") pod \"payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b\" (UID: \"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:57:33.800597 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.800441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b\" (UID: \"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:57:33.801521 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.801489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b\" (UID: \"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:57:33.809829 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.809682    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nlk\" (UniqueName: \"kubernetes.io/projected/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-kube-api-access-s9nlk\") pod \"payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b\" (UID: \"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:57:33.872289 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.872250    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:57:33.894654 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.894627    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"]
Jul 23 22:57:33.897014 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:57:33.896987    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bfcb41_50de_4730_8519_8ba50c677131.slice/crio-5cd67c3049fecd182c2a803cd407261c5d18c80585ed5e332a06212eebc83ca3 WatchSource:0}: Error finding container 5cd67c3049fecd182c2a803cd407261c5d18c80585ed5e332a06212eebc83ca3: Status 404 returned error can't find the container with id 5cd67c3049fecd182c2a803cd407261c5d18c80585ed5e332a06212eebc83ca3
Jul 23 22:57:33.898521 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.898500    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 22:57:33.911780 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:33.911749    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:57:34.021017 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.020982    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"]
Jul 23 22:57:34.023006 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:57:34.022973    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcbb03f_1a82_476b_8db4_6f95117625a6.slice/crio-4aba2886c1484d74e2a221fa1a8d713642ae9bf7a87625df7ff44722144f8c8b WatchSource:0}: Error finding container 4aba2886c1484d74e2a221fa1a8d713642ae9bf7a87625df7ff44722144f8c8b: Status 404 returned error can't find the container with id 4aba2886c1484d74e2a221fa1a8d713642ae9bf7a87625df7ff44722144f8c8b
Jul 23 22:57:34.069228 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.069205    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"]
Jul 23 22:57:34.071777 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:57:34.071752    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf157dee_e5dc_4cc4_9fc8_660fd5d0fc4a.slice/crio-a47ed92304204485c50869a6f5ee34095318167d98f441b748343d6e9de46279 WatchSource:0}: Error finding container a47ed92304204485c50869a6f5ee34095318167d98f441b748343d6e9de46279: Status 404 returned error can't find the container with id a47ed92304204485c50869a6f5ee34095318167d98f441b748343d6e9de46279
Jul 23 22:57:34.797648 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.797552    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b" event={"ID":"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a","Type":"ContainerStarted","Data":"e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0"}
Jul 23 22:57:34.797814 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.797705    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:57:34.797814 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.797744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b" event={"ID":"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a","Type":"ContainerStarted","Data":"a47ed92304204485c50869a6f5ee34095318167d98f441b748343d6e9de46279"}
Jul 23 22:57:34.798945 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.798903    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g" event={"ID":"ddcbb03f-1a82-476b-8db4-6f95117625a6","Type":"ContainerStarted","Data":"95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2"}
Jul 23 22:57:34.799045 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.798968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g" event={"ID":"ddcbb03f-1a82-476b-8db4-6f95117625a6","Type":"ContainerStarted","Data":"4aba2886c1484d74e2a221fa1a8d713642ae9bf7a87625df7ff44722144f8c8b"}
Jul 23 22:57:34.799045 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.799019    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:57:34.800486 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.800463    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r" event={"ID":"68bfcb41-50de-4730-8519-8ba50c677131","Type":"ContainerStarted","Data":"2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e"}
Jul 23 22:57:34.800581 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.800505    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r" event={"ID":"68bfcb41-50de-4730-8519-8ba50c677131","Type":"ContainerStarted","Data":"5cd67c3049fecd182c2a803cd407261c5d18c80585ed5e332a06212eebc83ca3"}
Jul 23 22:57:34.800581 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.800528    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:34.814679 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.814630    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b" podStartSLOduration=1.814616222 podStartE2EDuration="1.814616222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:57:34.813065033 +0000 UTC m=+2109.130593842" watchObservedRunningTime="2026-07-23 22:57:34.814616222 +0000 UTC m=+2109.132145032"
Jul 23 22:57:34.828608 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.828564    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g" podStartSLOduration=1.828551424 podStartE2EDuration="1.828551424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:57:34.827976105 +0000 UTC m=+2109.145504913" watchObservedRunningTime="2026-07-23 22:57:34.828551424 +0000 UTC m=+2109.146080261"
Jul 23 22:57:34.845370 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:34.845318    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r" podStartSLOduration=2.337478956 podStartE2EDuration="2.845298062s" totalImagesPullingTime="507.819106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:57:34.843748678 +0000 UTC m=+2109.161277491" watchObservedRunningTime="2026-07-23 22:57:34.845298062 +0000 UTC m=+2109.162826877"
Jul 23 22:57:40.686411 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.686369    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"]
Jul 23 22:57:40.697634 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.697575    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:40.700273 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.700100    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-798605-serving-cert\""
Jul 23 22:57:40.700564 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.700535    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"]
Jul 23 22:57:40.764489 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.764455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26jf4\" (UniqueName: \"kubernetes.io/projected/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-kube-api-access-26jf4\") pod \"maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:40.764631 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.764498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-maas-api-tls\") pod \"maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:40.764631 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.764587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-openshift-service-ca\") pod \"maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:40.810096 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.810065    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:57:40.865644 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.865609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-maas-api-tls\") pod \"maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:40.865830 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.865732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-openshift-service-ca\") pod \"maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:40.865939 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.865900    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26jf4\" (UniqueName: \"kubernetes.io/projected/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-kube-api-access-26jf4\") pod \"maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:40.866417 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.866392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-openshift-service-ca\") pod \"maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:40.868214 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.868192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-maas-api-tls\") pod \"maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:40.874022 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:40.874004    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26jf4\" (UniqueName: \"kubernetes.io/projected/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-kube-api-access-26jf4\") pod \"maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:41.009941 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.009838    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:41.143313 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.143282    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"]
Jul 23 22:57:41.146970 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:57:41.146803    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528d4b6b_5f0a_43e2_9eb7_7c065d4a2197.slice/crio-dabb78897be74865423e2fbb9207974b533ffa795febcca5204ebb0957a18066 WatchSource:0}: Error finding container dabb78897be74865423e2fbb9207974b533ffa795febcca5204ebb0957a18066: Status 404 returned error can't find the container with id dabb78897be74865423e2fbb9207974b533ffa795febcca5204ebb0957a18066
Jul 23 22:57:41.330758 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.330723    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"]
Jul 23 22:57:41.335402 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.335384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 22:57:41.337785 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.337764    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-798605\""
Jul 23 22:57:41.337870 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.337767    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-798605-dockercfg-fqttw\""
Jul 23 22:57:41.345119 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.345094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"]
Jul 23 22:57:41.366072 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.366038    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"]
Jul 23 22:57:41.370055 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.370034    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 22:57:41.371981 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.371954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lbqp\" (UniqueName: \"kubernetes.io/projected/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-kube-api-access-2lbqp\") pod \"payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n\" (UID: \"11cba5fd-a12e-4cce-9686-0ab7bbc467f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 22:57:41.372066 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.372009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n\" (UID: \"11cba5fd-a12e-4cce-9686-0ab7bbc467f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 22:57:41.378450 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.378427    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"]
Jul 23 22:57:41.472671 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.472630    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtlx4\" (UniqueName: \"kubernetes.io/projected/8212d36f-0808-4826-a9f1-35f00edb9cb8-kube-api-access-rtlx4\") pod \"payload-processing-e2e-inf-b-798605-5567d478b-4lhw5\" (UID: \"8212d36f-0808-4826-a9f1-35f00edb9cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 22:57:41.472888 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.472761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lbqp\" (UniqueName: \"kubernetes.io/projected/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-kube-api-access-2lbqp\") pod \"payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n\" (UID: \"11cba5fd-a12e-4cce-9686-0ab7bbc467f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 22:57:41.472888 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.472810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8212d36f-0808-4826-a9f1-35f00edb9cb8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-798605-5567d478b-4lhw5\" (UID: \"8212d36f-0808-4826-a9f1-35f00edb9cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 22:57:41.472888 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.472857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n\" (UID: \"11cba5fd-a12e-4cce-9686-0ab7bbc467f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 22:57:41.473772 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.473750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n\" (UID: \"11cba5fd-a12e-4cce-9686-0ab7bbc467f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 22:57:41.481347 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.481316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lbqp\" (UniqueName: \"kubernetes.io/projected/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-kube-api-access-2lbqp\") pod \"payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n\" (UID: \"11cba5fd-a12e-4cce-9686-0ab7bbc467f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 22:57:41.573372 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.573348    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtlx4\" (UniqueName: \"kubernetes.io/projected/8212d36f-0808-4826-a9f1-35f00edb9cb8-kube-api-access-rtlx4\") pod \"payload-processing-e2e-inf-b-798605-5567d478b-4lhw5\" (UID: \"8212d36f-0808-4826-a9f1-35f00edb9cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 22:57:41.573469 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.573408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8212d36f-0808-4826-a9f1-35f00edb9cb8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-798605-5567d478b-4lhw5\" (UID: \"8212d36f-0808-4826-a9f1-35f00edb9cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 22:57:41.574066 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.574047    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8212d36f-0808-4826-a9f1-35f00edb9cb8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-798605-5567d478b-4lhw5\" (UID: \"8212d36f-0808-4826-a9f1-35f00edb9cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 22:57:41.581317 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.581290    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtlx4\" (UniqueName: \"kubernetes.io/projected/8212d36f-0808-4826-a9f1-35f00edb9cb8-kube-api-access-rtlx4\") pod \"payload-processing-e2e-inf-b-798605-5567d478b-4lhw5\" (UID: \"8212d36f-0808-4826-a9f1-35f00edb9cb8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 22:57:41.647504 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.647473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 22:57:41.681690 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.681647    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 22:57:41.835561 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.835524    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v" event={"ID":"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197","Type":"ContainerStarted","Data":"54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd"}
Jul 23 22:57:41.836008 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.835593    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v" event={"ID":"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197","Type":"ContainerStarted","Data":"dabb78897be74865423e2fbb9207974b533ffa795febcca5204ebb0957a18066"}
Jul 23 22:57:41.836008 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.835711    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:41.853632 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.853580    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v" podStartSLOduration=1.464534348 podStartE2EDuration="1.853564352s" totalImagesPullingTime="389.030004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:57:41.850826482 +0000 UTC m=+2116.168355303" watchObservedRunningTime="2026-07-23 22:57:41.853564352 +0000 UTC m=+2116.171093209"
Jul 23 22:57:41.997656 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:41.997632    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"]
Jul 23 22:57:41.999470 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:57:41.999441    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cba5fd_a12e_4cce_9686_0ab7bbc467f1.slice/crio-e85c44b738d3a5a40f26e7fcb995cf5143bb8c53d8fd72d4d893c84ea903a280 WatchSource:0}: Error finding container e85c44b738d3a5a40f26e7fcb995cf5143bb8c53d8fd72d4d893c84ea903a280: Status 404 returned error can't find the container with id e85c44b738d3a5a40f26e7fcb995cf5143bb8c53d8fd72d4d893c84ea903a280
Jul 23 22:57:42.035380 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:42.035233    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"]
Jul 23 22:57:42.038170 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:57:42.038140    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8212d36f_0808_4826_a9f1_35f00edb9cb8.slice/crio-c7023b87444de79ac1d6a3e505e20f5984cb7f04449c8367e858b6f7a592bb18 WatchSource:0}: Error finding container c7023b87444de79ac1d6a3e505e20f5984cb7f04449c8367e858b6f7a592bb18: Status 404 returned error can't find the container with id c7023b87444de79ac1d6a3e505e20f5984cb7f04449c8367e858b6f7a592bb18
Jul 23 22:57:42.847668 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:42.847631    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5" event={"ID":"8212d36f-0808-4826-a9f1-35f00edb9cb8","Type":"ContainerStarted","Data":"73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04"}
Jul 23 22:57:42.848159 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:42.847696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5" event={"ID":"8212d36f-0808-4826-a9f1-35f00edb9cb8","Type":"ContainerStarted","Data":"c7023b87444de79ac1d6a3e505e20f5984cb7f04449c8367e858b6f7a592bb18"}
Jul 23 22:57:42.848159 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:42.847775    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 22:57:42.849346 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:42.849323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n" event={"ID":"11cba5fd-a12e-4cce-9686-0ab7bbc467f1","Type":"ContainerStarted","Data":"f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e"}
Jul 23 22:57:42.849456 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:42.849360    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n" event={"ID":"11cba5fd-a12e-4cce-9686-0ab7bbc467f1","Type":"ContainerStarted","Data":"e85c44b738d3a5a40f26e7fcb995cf5143bb8c53d8fd72d4d893c84ea903a280"}
Jul 23 22:57:42.849456 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:42.849415    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 22:57:42.868313 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:42.868274    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5" podStartSLOduration=1.86826093 podStartE2EDuration="1.86826093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:57:42.866304707 +0000 UTC m=+2117.183833517" watchObservedRunningTime="2026-07-23 22:57:42.86826093 +0000 UTC m=+2117.185789739"
Jul 23 22:57:42.882312 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:42.882272    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n" podStartSLOduration=1.8822600779999998 podStartE2EDuration="1.882260078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:57:42.881450986 +0000 UTC m=+2117.198979807" watchObservedRunningTime="2026-07-23 22:57:42.882260078 +0000 UTC m=+2117.199788887"
Jul 23 22:57:45.806647 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:45.806617    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:57:45.807097 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:45.806689    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:57:47.854887 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:47.854852    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 22:57:53.856953 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:53.856888    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 22:57:53.857431 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:57:53.857001    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 22:59:31.590407 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:31.590369    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"]
Jul 23 22:59:31.594003 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:31.593983    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"
Jul 23 22:59:31.596603 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:31.596581    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6cq4\""
Jul 23 22:59:31.601875 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:31.601850    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"]
Jul 23 22:59:31.712984 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:31.712950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab332a2-2653-488c-93de-403536dd7b4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh\" (UID: \"dab332a2-2653-488c-93de-403536dd7b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"
Jul 23 22:59:31.813476 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:31.813435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab332a2-2653-488c-93de-403536dd7b4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh\" (UID: \"dab332a2-2653-488c-93de-403536dd7b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"
Jul 23 22:59:31.814134 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:31.814113    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab332a2-2653-488c-93de-403536dd7b4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh\" (UID: \"dab332a2-2653-488c-93de-403536dd7b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"
Jul 23 22:59:31.905665 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:31.905572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"
Jul 23 22:59:32.052392 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:32.052363    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"]
Jul 23 22:59:32.055794 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:59:32.055768    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab332a2_2653_488c_93de_403536dd7b4c.slice/crio-b3cd7c00983cf75a09c9c5802cf8a2b3fd8fa41bf57e62a9bf923ff1c1882de6 WatchSource:0}: Error finding container b3cd7c00983cf75a09c9c5802cf8a2b3fd8fa41bf57e62a9bf923ff1c1882de6: Status 404 returned error can't find the container with id b3cd7c00983cf75a09c9c5802cf8a2b3fd8fa41bf57e62a9bf923ff1c1882de6
Jul 23 22:59:32.369783 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:32.369742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh" event={"ID":"dab332a2-2653-488c-93de-403536dd7b4c","Type":"ContainerStarted","Data":"46369b3bb61f94e561b35dc62c71ecc2cb7e44381e0d94289e9cc9b1bc9c7485"}
Jul 23 22:59:32.369989 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:32.369807    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh" event={"ID":"dab332a2-2653-488c-93de-403536dd7b4c","Type":"ContainerStarted","Data":"b3cd7c00983cf75a09c9c5802cf8a2b3fd8fa41bf57e62a9bf923ff1c1882de6"}
Jul 23 22:59:32.387828 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:32.387772    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh" podStartSLOduration=1.38775573 podStartE2EDuration="1.38775573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:59:32.386618837 +0000 UTC m=+2226.704147646" watchObservedRunningTime="2026-07-23 22:59:32.38775573 +0000 UTC m=+2226.705284538"
Jul 23 22:59:33.375814 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:33.375777    2583 generic.go:356] "Generic (PLEG): container finished" podID="dab332a2-2653-488c-93de-403536dd7b4c" containerID="46369b3bb61f94e561b35dc62c71ecc2cb7e44381e0d94289e9cc9b1bc9c7485" exitCode=0
Jul 23 22:59:33.376237 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:33.375827    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh" event={"ID":"dab332a2-2653-488c-93de-403536dd7b4c","Type":"ContainerDied","Data":"46369b3bb61f94e561b35dc62c71ecc2cb7e44381e0d94289e9cc9b1bc9c7485"}
Jul 23 22:59:34.515231 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:34.515206    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"
Jul 23 22:59:34.637864 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:34.637776    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab332a2-2653-488c-93de-403536dd7b4c-maas-api-service-ca\") pod \"dab332a2-2653-488c-93de-403536dd7b4c\" (UID: \"dab332a2-2653-488c-93de-403536dd7b4c\") "
Jul 23 22:59:34.638134 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:34.638112    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab332a2-2653-488c-93de-403536dd7b4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dab332a2-2653-488c-93de-403536dd7b4c" (UID: "dab332a2-2653-488c-93de-403536dd7b4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:59:34.739218 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:34.739185    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab332a2-2653-488c-93de-403536dd7b4c-maas-api-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:59:35.387716 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:35.387678    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh" event={"ID":"dab332a2-2653-488c-93de-403536dd7b4c","Type":"ContainerDied","Data":"b3cd7c00983cf75a09c9c5802cf8a2b3fd8fa41bf57e62a9bf923ff1c1882de6"}
Jul 23 22:59:35.387890 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:35.387728    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3cd7c00983cf75a09c9c5802cf8a2b3fd8fa41bf57e62a9bf923ff1c1882de6"
Jul 23 22:59:35.387890 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:35.387693    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"
Jul 23 22:59:41.955526 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:41.955494    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"]
Jul 23 22:59:41.956021 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:41.955689    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r" podUID="68bfcb41-50de-4730-8519-8ba50c677131" containerName="maas-api" containerID="cri-o://2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e" gracePeriod=30
Jul 23 22:59:41.996415 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:41.996389    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"]
Jul 23 22:59:41.997126 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:41.996592    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b" podUID="af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a" containerName="payload-processing" containerID="cri-o://e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0" gracePeriod=30
Jul 23 22:59:42.004469 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.004438    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"]
Jul 23 22:59:42.004672 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.004650    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g" podUID="ddcbb03f-1a82-476b-8db4-6f95117625a6" containerName="payload-pre-processing" containerID="cri-o://95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2" gracePeriod=30
Jul 23 22:59:42.267425 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.267395    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:59:42.313316 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.313293    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:59:42.317044 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.317019    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:59:42.408824 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.408783    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68bfcb41-50de-4730-8519-8ba50c677131-maas-api-tls\") pod \"68bfcb41-50de-4730-8519-8ba50c677131\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") "
Jul 23 22:59:42.408824 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.408828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8xbf\" (UniqueName: \"kubernetes.io/projected/68bfcb41-50de-4730-8519-8ba50c677131-kube-api-access-v8xbf\") pod \"68bfcb41-50de-4730-8519-8ba50c677131\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") "
Jul 23 22:59:42.409068 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.408870    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpn5t\" (UniqueName: \"kubernetes.io/projected/ddcbb03f-1a82-476b-8db4-6f95117625a6-kube-api-access-wpn5t\") pod \"ddcbb03f-1a82-476b-8db4-6f95117625a6\" (UID: \"ddcbb03f-1a82-476b-8db4-6f95117625a6\") "
Jul 23 22:59:42.409068 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.408889    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bfcb41-50de-4730-8519-8ba50c677131-openshift-service-ca\") pod \"68bfcb41-50de-4730-8519-8ba50c677131\" (UID: \"68bfcb41-50de-4730-8519-8ba50c677131\") "
Jul 23 22:59:42.409068 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.408952    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcbb03f-1a82-476b-8db4-6f95117625a6-plugins-config-volume\") pod \"ddcbb03f-1a82-476b-8db4-6f95117625a6\" (UID: \"ddcbb03f-1a82-476b-8db4-6f95117625a6\") "
Jul 23 22:59:42.409068 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.408975    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9nlk\" (UniqueName: \"kubernetes.io/projected/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-kube-api-access-s9nlk\") pod \"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a\" (UID: \"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a\") "
Jul 23 22:59:42.409247 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.409067    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-plugins-config-volume\") pod \"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a\" (UID: \"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a\") "
Jul 23 22:59:42.409375 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.409339    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bfcb41-50de-4730-8519-8ba50c677131-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68bfcb41-50de-4730-8519-8ba50c677131" (UID: "68bfcb41-50de-4730-8519-8ba50c677131"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:59:42.409497 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.409472    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddcbb03f-1a82-476b-8db4-6f95117625a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ddcbb03f-1a82-476b-8db4-6f95117625a6" (UID: "ddcbb03f-1a82-476b-8db4-6f95117625a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:59:42.409769 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.409741    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a" (UID: "af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:59:42.411098 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.411071    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bfcb41-50de-4730-8519-8ba50c677131-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68bfcb41-50de-4730-8519-8ba50c677131" (UID: "68bfcb41-50de-4730-8519-8ba50c677131"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:59:42.411188 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.411095    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bfcb41-50de-4730-8519-8ba50c677131-kube-api-access-v8xbf" (OuterVolumeSpecName: "kube-api-access-v8xbf") pod "68bfcb41-50de-4730-8519-8ba50c677131" (UID: "68bfcb41-50de-4730-8519-8ba50c677131"). InnerVolumeSpecName "kube-api-access-v8xbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:59:42.411423 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.411397    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddcbb03f-1a82-476b-8db4-6f95117625a6-kube-api-access-wpn5t" (OuterVolumeSpecName: "kube-api-access-wpn5t") pod "ddcbb03f-1a82-476b-8db4-6f95117625a6" (UID: "ddcbb03f-1a82-476b-8db4-6f95117625a6"). InnerVolumeSpecName "kube-api-access-wpn5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:59:42.411488 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.411422    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-kube-api-access-s9nlk" (OuterVolumeSpecName: "kube-api-access-s9nlk") pod "af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a" (UID: "af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a"). InnerVolumeSpecName "kube-api-access-s9nlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:59:42.430831 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.430803    2583 generic.go:356] "Generic (PLEG): container finished" podID="af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a" containerID="e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0" exitCode=0
Jul 23 22:59:42.430960 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.430868    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"
Jul 23 22:59:42.430960 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.430886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b" event={"ID":"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a","Type":"ContainerDied","Data":"e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0"}
Jul 23 22:59:42.431068 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.430986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b" event={"ID":"af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a","Type":"ContainerDied","Data":"a47ed92304204485c50869a6f5ee34095318167d98f441b748343d6e9de46279"}
Jul 23 22:59:42.431068 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.431011    2583 scope.go:117] "RemoveContainer" containerID="e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0"
Jul 23 22:59:42.432237 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.432215    2583 generic.go:356] "Generic (PLEG): container finished" podID="ddcbb03f-1a82-476b-8db4-6f95117625a6" containerID="95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2" exitCode=0
Jul 23 22:59:42.432317 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.432265    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"
Jul 23 22:59:42.432317 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.432296    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g" event={"ID":"ddcbb03f-1a82-476b-8db4-6f95117625a6","Type":"ContainerDied","Data":"95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2"}
Jul 23 22:59:42.432429 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.432334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g" event={"ID":"ddcbb03f-1a82-476b-8db4-6f95117625a6","Type":"ContainerDied","Data":"4aba2886c1484d74e2a221fa1a8d713642ae9bf7a87625df7ff44722144f8c8b"}
Jul 23 22:59:42.433772 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.433741    2583 generic.go:356] "Generic (PLEG): container finished" podID="68bfcb41-50de-4730-8519-8ba50c677131" containerID="2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e" exitCode=0
Jul 23 22:59:42.433895 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.433796    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"
Jul 23 22:59:42.433895 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.433809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r" event={"ID":"68bfcb41-50de-4730-8519-8ba50c677131","Type":"ContainerDied","Data":"2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e"}
Jul 23 22:59:42.433895 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.433848    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r" event={"ID":"68bfcb41-50de-4730-8519-8ba50c677131","Type":"ContainerDied","Data":"5cd67c3049fecd182c2a803cd407261c5d18c80585ed5e332a06212eebc83ca3"}
Jul 23 22:59:42.440693 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.440637    2583 scope.go:117] "RemoveContainer" containerID="e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0"
Jul 23 22:59:42.441230 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:59:42.440982    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0\": container with ID starting with e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0 not found: ID does not exist" containerID="e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0"
Jul 23 22:59:42.441230 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.441019    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0"} err="failed to get container status \"e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0\": rpc error: code = NotFound desc = could not find container \"e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0\": container with ID starting with e5a3375c8909935aca1da72ce3865a50e6aa8aedef6abf6491c284421e6b97a0 not found: ID does not exist"
Jul 23 22:59:42.441230 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.441040    2583 scope.go:117] "RemoveContainer" containerID="95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2"
Jul 23 22:59:42.449334 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.449318    2583 scope.go:117] "RemoveContainer" containerID="95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2"
Jul 23 22:59:42.449551 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:59:42.449533    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2\": container with ID starting with 95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2 not found: ID does not exist" containerID="95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2"
Jul 23 22:59:42.449599 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.449558    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2"} err="failed to get container status \"95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2\": rpc error: code = NotFound desc = could not find container \"95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2\": container with ID starting with 95237223aded967cf6707da3ecb1e43fdce2ccc8c476b9166dfef027ec87f7e2 not found: ID does not exist"
Jul 23 22:59:42.449599 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.449575    2583 scope.go:117] "RemoveContainer" containerID="2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e"
Jul 23 22:59:42.456169 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.456147    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"]
Jul 23 22:59:42.458734 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.458716    2583 scope.go:117] "RemoveContainer" containerID="2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e"
Jul 23 22:59:42.459023 ip-10-0-133-131 kubenswrapper[2583]: E0723 22:59:42.459001    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e\": container with ID starting with 2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e not found: ID does not exist" containerID="2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e"
Jul 23 22:59:42.459092 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.459027    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e"} err="failed to get container status \"2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e\": rpc error: code = NotFound desc = could not find container \"2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e\": container with ID starting with 2c5d822b38c0091495f757c784a7e2b7fd31afd4b316d3d563db8d5e90e7691e not found: ID does not exist"
Jul 23 22:59:42.460041 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.460023    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70c7c-5965d876b8-24k8b"]
Jul 23 22:59:42.469374 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.469350    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"]
Jul 23 22:59:42.473358 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.473335    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70c7c-85655c94f9-d6n6r"]
Jul 23 22:59:42.482596 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.482573    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"]
Jul 23 22:59:42.487492 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.487443    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70c7c-8d7d85944-p6l4g"]
Jul 23 22:59:42.510572 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.510545    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpn5t\" (UniqueName: \"kubernetes.io/projected/ddcbb03f-1a82-476b-8db4-6f95117625a6-kube-api-access-wpn5t\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:59:42.510572 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.510570    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bfcb41-50de-4730-8519-8ba50c677131-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:59:42.510707 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.510585    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcbb03f-1a82-476b-8db4-6f95117625a6-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:59:42.510707 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.510599    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nlk\" (UniqueName: \"kubernetes.io/projected/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-kube-api-access-s9nlk\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:59:42.510707 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.510612    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:59:42.510707 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.510625    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68bfcb41-50de-4730-8519-8ba50c677131-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:59:42.510707 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:42.510640    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8xbf\" (UniqueName: \"kubernetes.io/projected/68bfcb41-50de-4730-8519-8ba50c677131-kube-api-access-v8xbf\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:59:44.229248 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:44.229214    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68bfcb41-50de-4730-8519-8ba50c677131" path="/var/lib/kubelet/pods/68bfcb41-50de-4730-8519-8ba50c677131/volumes"
Jul 23 22:59:44.229613 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:44.229555    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a" path="/var/lib/kubelet/pods/af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a/volumes"
Jul 23 22:59:44.229871 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:44.229859    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddcbb03f-1a82-476b-8db4-6f95117625a6" path="/var/lib/kubelet/pods/ddcbb03f-1a82-476b-8db4-6f95117625a6/volumes"
Jul 23 22:59:46.672914 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:46.672869    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"]
Jul 23 22:59:46.676687 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:46.676661    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70c7c-78fsh"]
Jul 23 22:59:48.229016 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:48.228982    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab332a2-2653-488c-93de-403536dd7b4c" path="/var/lib/kubelet/pods/dab332a2-2653-488c-93de-403536dd7b4c/volumes"
Jul 23 22:59:53.614912 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.614874    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"]
Jul 23 22:59:53.615310 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615247    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab332a2-2653-488c-93de-403536dd7b4c" containerName="revoke-keys"
Jul 23 22:59:53.615310 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615259    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab332a2-2653-488c-93de-403536dd7b4c" containerName="revoke-keys"
Jul 23 22:59:53.615310 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615267    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a" containerName="payload-processing"
Jul 23 22:59:53.615310 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615274    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a" containerName="payload-processing"
Jul 23 22:59:53.615310 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615285    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bfcb41-50de-4730-8519-8ba50c677131" containerName="maas-api"
Jul 23 22:59:53.615310 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615291    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bfcb41-50de-4730-8519-8ba50c677131" containerName="maas-api"
Jul 23 22:59:53.615310 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615304    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddcbb03f-1a82-476b-8db4-6f95117625a6" containerName="payload-pre-processing"
Jul 23 22:59:53.615310 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615309    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddcbb03f-1a82-476b-8db4-6f95117625a6" containerName="payload-pre-processing"
Jul 23 22:59:53.615557 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615364    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="af157dee-e5dc-4cc4-9fc8-660fd5d0fc4a" containerName="payload-processing"
Jul 23 22:59:53.615557 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615372    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddcbb03f-1a82-476b-8db4-6f95117625a6" containerName="payload-pre-processing"
Jul 23 22:59:53.615557 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615378    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bfcb41-50de-4730-8519-8ba50c677131" containerName="maas-api"
Jul 23 22:59:53.615557 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.615388    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab332a2-2653-488c-93de-403536dd7b4c" containerName="revoke-keys"
Jul 23 22:59:53.620142 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.620124    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"
Jul 23 22:59:53.622559 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.622534    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6cq4\""
Jul 23 22:59:53.626309 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.626253    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"]
Jul 23 22:59:53.697745 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.697713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96c7537-0aa8-4e03-9a03-ec7bf44ece98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-798605-mfqzn\" (UID: \"c96c7537-0aa8-4e03-9a03-ec7bf44ece98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"
Jul 23 22:59:53.799199 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.799158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96c7537-0aa8-4e03-9a03-ec7bf44ece98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-798605-mfqzn\" (UID: \"c96c7537-0aa8-4e03-9a03-ec7bf44ece98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"
Jul 23 22:59:53.799772 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.799749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96c7537-0aa8-4e03-9a03-ec7bf44ece98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-798605-mfqzn\" (UID: \"c96c7537-0aa8-4e03-9a03-ec7bf44ece98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"
Jul 23 22:59:53.932783 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:53.932690    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"
Jul 23 22:59:54.064798 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:54.064769    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"]
Jul 23 22:59:54.067039 ip-10-0-133-131 kubenswrapper[2583]: W0723 22:59:54.067000    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96c7537_0aa8_4e03_9a03_ec7bf44ece98.slice/crio-6748d02e7b6613cf64a4ebe21281d2d540158c79f09574d2efddcd4b29c5d214 WatchSource:0}: Error finding container 6748d02e7b6613cf64a4ebe21281d2d540158c79f09574d2efddcd4b29c5d214: Status 404 returned error can't find the container with id 6748d02e7b6613cf64a4ebe21281d2d540158c79f09574d2efddcd4b29c5d214
Jul 23 22:59:54.495146 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:54.495109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn" event={"ID":"c96c7537-0aa8-4e03-9a03-ec7bf44ece98","Type":"ContainerStarted","Data":"0aed4b8cb1fa2b43c317fa1b61875bfffc0e106d2bbb49dde2c04b215a9ee567"}
Jul 23 22:59:54.495304 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:54.495167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn" event={"ID":"c96c7537-0aa8-4e03-9a03-ec7bf44ece98","Type":"ContainerStarted","Data":"6748d02e7b6613cf64a4ebe21281d2d540158c79f09574d2efddcd4b29c5d214"}
Jul 23 22:59:54.510126 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:54.510075    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn" podStartSLOduration=1.5100573769999999 podStartE2EDuration="1.510057377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:59:54.509790899 +0000 UTC m=+2248.827319709" watchObservedRunningTime="2026-07-23 22:59:54.510057377 +0000 UTC m=+2248.827586169"
Jul 23 22:59:55.501420 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:55.501384    2583 generic.go:356] "Generic (PLEG): container finished" podID="c96c7537-0aa8-4e03-9a03-ec7bf44ece98" containerID="0aed4b8cb1fa2b43c317fa1b61875bfffc0e106d2bbb49dde2c04b215a9ee567" exitCode=0
Jul 23 22:59:55.501869 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:55.501470    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn" event={"ID":"c96c7537-0aa8-4e03-9a03-ec7bf44ece98","Type":"ContainerDied","Data":"0aed4b8cb1fa2b43c317fa1b61875bfffc0e106d2bbb49dde2c04b215a9ee567"}
Jul 23 22:59:56.640819 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:56.640797    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"
Jul 23 22:59:56.723400 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:56.723369    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96c7537-0aa8-4e03-9a03-ec7bf44ece98-maas-api-service-ca\") pod \"c96c7537-0aa8-4e03-9a03-ec7bf44ece98\" (UID: \"c96c7537-0aa8-4e03-9a03-ec7bf44ece98\") "
Jul 23 22:59:56.723721 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:56.723701    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96c7537-0aa8-4e03-9a03-ec7bf44ece98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c96c7537-0aa8-4e03-9a03-ec7bf44ece98" (UID: "c96c7537-0aa8-4e03-9a03-ec7bf44ece98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:59:56.824520 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:56.824492    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96c7537-0aa8-4e03-9a03-ec7bf44ece98-maas-api-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 22:59:57.512646 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:57.512611    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"
Jul 23 22:59:57.512812 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:57.512639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn" event={"ID":"c96c7537-0aa8-4e03-9a03-ec7bf44ece98","Type":"ContainerDied","Data":"6748d02e7b6613cf64a4ebe21281d2d540158c79f09574d2efddcd4b29c5d214"}
Jul 23 22:59:57.512812 ip-10-0-133-131 kubenswrapper[2583]: I0723 22:59:57.512692    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6748d02e7b6613cf64a4ebe21281d2d540158c79f09574d2efddcd4b29c5d214"
Jul 23 23:00:03.984897 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:03.984848    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"]
Jul 23 23:00:03.985876 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:03.985838    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v" podUID="528d4b6b-5f0a-43e2-9eb7-7c065d4a2197" containerName="maas-api" containerID="cri-o://54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd" gracePeriod=30
Jul 23 23:00:04.050553 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.050518    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"]
Jul 23 23:00:04.050838 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.050785    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5" podUID="8212d36f-0808-4826-a9f1-35f00edb9cb8" containerName="payload-processing" containerID="cri-o://73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04" gracePeriod=30
Jul 23 23:00:04.061761 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.061734    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"]
Jul 23 23:00:04.062043 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.061978    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n" podUID="11cba5fd-a12e-4cce-9686-0ab7bbc467f1" containerName="payload-pre-processing" containerID="cri-o://f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e" gracePeriod=30
Jul 23 23:00:04.364211 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.364180    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 23:00:04.378755 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.378724    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 23:00:04.380999 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.380980    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 23:00:04.487270 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.487235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-plugins-config-volume\") pod \"11cba5fd-a12e-4cce-9686-0ab7bbc467f1\" (UID: \"11cba5fd-a12e-4cce-9686-0ab7bbc467f1\") "
Jul 23 23:00:04.487477 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.487276    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lbqp\" (UniqueName: \"kubernetes.io/projected/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-kube-api-access-2lbqp\") pod \"11cba5fd-a12e-4cce-9686-0ab7bbc467f1\" (UID: \"11cba5fd-a12e-4cce-9686-0ab7bbc467f1\") "
Jul 23 23:00:04.487477 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.487299    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8212d36f-0808-4826-a9f1-35f00edb9cb8-plugins-config-volume\") pod \"8212d36f-0808-4826-a9f1-35f00edb9cb8\" (UID: \"8212d36f-0808-4826-a9f1-35f00edb9cb8\") "
Jul 23 23:00:04.487477 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.487326    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-maas-api-tls\") pod \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") "
Jul 23 23:00:04.487477 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.487374    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtlx4\" (UniqueName: \"kubernetes.io/projected/8212d36f-0808-4826-a9f1-35f00edb9cb8-kube-api-access-rtlx4\") pod \"8212d36f-0808-4826-a9f1-35f00edb9cb8\" (UID: \"8212d36f-0808-4826-a9f1-35f00edb9cb8\") "
Jul 23 23:00:04.487477 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.487397    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26jf4\" (UniqueName: \"kubernetes.io/projected/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-kube-api-access-26jf4\") pod \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") "
Jul 23 23:00:04.487477 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.487455    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-openshift-service-ca\") pod \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\" (UID: \"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197\") "
Jul 23 23:00:04.487814 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.487644    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8212d36f-0808-4826-a9f1-35f00edb9cb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8212d36f-0808-4826-a9f1-35f00edb9cb8" (UID: "8212d36f-0808-4826-a9f1-35f00edb9cb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:00:04.487814 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.487693    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11cba5fd-a12e-4cce-9686-0ab7bbc467f1" (UID: "11cba5fd-a12e-4cce-9686-0ab7bbc467f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:00:04.487814 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.487760    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8212d36f-0808-4826-a9f1-35f00edb9cb8-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:04.488040 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.488013    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "528d4b6b-5f0a-43e2-9eb7-7c065d4a2197" (UID: "528d4b6b-5f0a-43e2-9eb7-7c065d4a2197"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:00:04.489647 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.489624    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "528d4b6b-5f0a-43e2-9eb7-7c065d4a2197" (UID: "528d4b6b-5f0a-43e2-9eb7-7c065d4a2197"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 23:00:04.489750 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.489659    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8212d36f-0808-4826-a9f1-35f00edb9cb8-kube-api-access-rtlx4" (OuterVolumeSpecName: "kube-api-access-rtlx4") pod "8212d36f-0808-4826-a9f1-35f00edb9cb8" (UID: "8212d36f-0808-4826-a9f1-35f00edb9cb8"). InnerVolumeSpecName "kube-api-access-rtlx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:00:04.489750 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.489672    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-kube-api-access-2lbqp" (OuterVolumeSpecName: "kube-api-access-2lbqp") pod "11cba5fd-a12e-4cce-9686-0ab7bbc467f1" (UID: "11cba5fd-a12e-4cce-9686-0ab7bbc467f1"). InnerVolumeSpecName "kube-api-access-2lbqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:00:04.489750 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.489672    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-kube-api-access-26jf4" (OuterVolumeSpecName: "kube-api-access-26jf4") pod "528d4b6b-5f0a-43e2-9eb7-7c065d4a2197" (UID: "528d4b6b-5f0a-43e2-9eb7-7c065d4a2197"). InnerVolumeSpecName "kube-api-access-26jf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:00:04.549448 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.549365    2583 generic.go:356] "Generic (PLEG): container finished" podID="528d4b6b-5f0a-43e2-9eb7-7c065d4a2197" containerID="54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd" exitCode=0
Jul 23 23:00:04.549448 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.549402    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"
Jul 23 23:00:04.549638 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.549446    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v" event={"ID":"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197","Type":"ContainerDied","Data":"54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd"}
Jul 23 23:00:04.549638 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.549504    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v" event={"ID":"528d4b6b-5f0a-43e2-9eb7-7c065d4a2197","Type":"ContainerDied","Data":"dabb78897be74865423e2fbb9207974b533ffa795febcca5204ebb0957a18066"}
Jul 23 23:00:04.549638 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.549537    2583 scope.go:117] "RemoveContainer" containerID="54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd"
Jul 23 23:00:04.550758 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.550741    2583 generic.go:356] "Generic (PLEG): container finished" podID="8212d36f-0808-4826-a9f1-35f00edb9cb8" containerID="73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04" exitCode=0
Jul 23 23:00:04.550848 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.550783    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"
Jul 23 23:00:04.550848 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.550786    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5" event={"ID":"8212d36f-0808-4826-a9f1-35f00edb9cb8","Type":"ContainerDied","Data":"73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04"}
Jul 23 23:00:04.550848 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.550831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5" event={"ID":"8212d36f-0808-4826-a9f1-35f00edb9cb8","Type":"ContainerDied","Data":"c7023b87444de79ac1d6a3e505e20f5984cb7f04449c8367e858b6f7a592bb18"}
Jul 23 23:00:04.552263 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.552232    2583 generic.go:356] "Generic (PLEG): container finished" podID="11cba5fd-a12e-4cce-9686-0ab7bbc467f1" containerID="f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e" exitCode=0
Jul 23 23:00:04.552392 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.552270    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"
Jul 23 23:00:04.552392 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.552274    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n" event={"ID":"11cba5fd-a12e-4cce-9686-0ab7bbc467f1","Type":"ContainerDied","Data":"f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e"}
Jul 23 23:00:04.552392 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.552322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n" event={"ID":"11cba5fd-a12e-4cce-9686-0ab7bbc467f1","Type":"ContainerDied","Data":"e85c44b738d3a5a40f26e7fcb995cf5143bb8c53d8fd72d4d893c84ea903a280"}
Jul 23 23:00:04.563575 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.563553    2583 scope.go:117] "RemoveContainer" containerID="54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd"
Jul 23 23:00:04.563918 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:00:04.563891    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd\": container with ID starting with 54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd not found: ID does not exist" containerID="54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd"
Jul 23 23:00:04.564042 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.563960    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd"} err="failed to get container status \"54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd\": rpc error: code = NotFound desc = could not find container \"54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd\": container with ID starting with 54a2f98d30aaf713cec584ce41f50fc8ef8cb1394116c79a8d0db0a06264fecd not found: ID does not exist"
Jul 23 23:00:04.564042 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.563985    2583 scope.go:117] "RemoveContainer" containerID="73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04"
Jul 23 23:00:04.571992 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.571976    2583 scope.go:117] "RemoveContainer" containerID="73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04"
Jul 23 23:00:04.572232 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:00:04.572213    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04\": container with ID starting with 73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04 not found: ID does not exist" containerID="73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04"
Jul 23 23:00:04.572306 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.572242    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04"} err="failed to get container status \"73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04\": rpc error: code = NotFound desc = could not find container \"73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04\": container with ID starting with 73d461ed37b32fabcb67511fadce15f84776a8784424b47f754f36c4524e5e04 not found: ID does not exist"
Jul 23 23:00:04.572306 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.572265    2583 scope.go:117] "RemoveContainer" containerID="f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e"
Jul 23 23:00:04.577301 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.577265    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"]
Jul 23 23:00:04.579994 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.579973    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-798605-69bfbb9f84-kzp2v"]
Jul 23 23:00:04.582167 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.582152    2583 scope.go:117] "RemoveContainer" containerID="f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e"
Jul 23 23:00:04.582398 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:00:04.582379    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e\": container with ID starting with f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e not found: ID does not exist" containerID="f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e"
Jul 23 23:00:04.582451 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.582406    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e"} err="failed to get container status \"f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e\": rpc error: code = NotFound desc = could not find container \"f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e\": container with ID starting with f5fef13c92c27a48877d8d13fa5d53269eee7f5e39a349c195b6e4a50af7576e not found: ID does not exist"
Jul 23 23:00:04.588392 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.588367    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:04.588479 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.588397    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtlx4\" (UniqueName: \"kubernetes.io/projected/8212d36f-0808-4826-a9f1-35f00edb9cb8-kube-api-access-rtlx4\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:04.588479 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.588413    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26jf4\" (UniqueName: \"kubernetes.io/projected/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-kube-api-access-26jf4\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:04.588479 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.588429    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:04.588479 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.588443    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:04.588479 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.588457    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lbqp\" (UniqueName: \"kubernetes.io/projected/11cba5fd-a12e-4cce-9686-0ab7bbc467f1-kube-api-access-2lbqp\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:04.588845 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.588829    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"]
Jul 23 23:00:04.592565 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.592547    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-798605-c4654d5cb-hd55n"]
Jul 23 23:00:04.603170 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.603145    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"]
Jul 23 23:00:04.607282 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:04.607260    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-798605-5567d478b-4lhw5"]
Jul 23 23:00:06.228437 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:06.228406    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11cba5fd-a12e-4cce-9686-0ab7bbc467f1" path="/var/lib/kubelet/pods/11cba5fd-a12e-4cce-9686-0ab7bbc467f1/volumes"
Jul 23 23:00:06.228803 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:06.228779    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="528d4b6b-5f0a-43e2-9eb7-7c065d4a2197" path="/var/lib/kubelet/pods/528d4b6b-5f0a-43e2-9eb7-7c065d4a2197/volumes"
Jul 23 23:00:06.229114 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:06.229101    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8212d36f-0808-4826-a9f1-35f00edb9cb8" path="/var/lib/kubelet/pods/8212d36f-0808-4826-a9f1-35f00edb9cb8/volumes"
Jul 23 23:00:08.683168 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:08.683123    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"]
Jul 23 23:00:08.684941 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:08.684901    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-798605-mfqzn"]
Jul 23 23:00:10.229173 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:10.229139    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c96c7537-0aa8-4e03-9a03-ec7bf44ece98" path="/var/lib/kubelet/pods/c96c7537-0aa8-4e03-9a03-ec7bf44ece98/volumes"
Jul 23 23:00:19.425447 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425408    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"]
Jul 23 23:00:19.425835 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425728    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="528d4b6b-5f0a-43e2-9eb7-7c065d4a2197" containerName="maas-api"
Jul 23 23:00:19.425835 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425740    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="528d4b6b-5f0a-43e2-9eb7-7c065d4a2197" containerName="maas-api"
Jul 23 23:00:19.425835 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425751    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cba5fd-a12e-4cce-9686-0ab7bbc467f1" containerName="payload-pre-processing"
Jul 23 23:00:19.425835 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425756    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cba5fd-a12e-4cce-9686-0ab7bbc467f1" containerName="payload-pre-processing"
Jul 23 23:00:19.425835 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425769    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c96c7537-0aa8-4e03-9a03-ec7bf44ece98" containerName="revoke-keys"
Jul 23 23:00:19.425835 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425774    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c96c7537-0aa8-4e03-9a03-ec7bf44ece98" containerName="revoke-keys"
Jul 23 23:00:19.425835 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425784    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8212d36f-0808-4826-a9f1-35f00edb9cb8" containerName="payload-processing"
Jul 23 23:00:19.425835 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425789    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8212d36f-0808-4826-a9f1-35f00edb9cb8" containerName="payload-processing"
Jul 23 23:00:19.426125 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425846    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c96c7537-0aa8-4e03-9a03-ec7bf44ece98" containerName="revoke-keys"
Jul 23 23:00:19.426125 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425855    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8212d36f-0808-4826-a9f1-35f00edb9cb8" containerName="payload-processing"
Jul 23 23:00:19.426125 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425864    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="528d4b6b-5f0a-43e2-9eb7-7c065d4a2197" containerName="maas-api"
Jul 23 23:00:19.426125 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.425870    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="11cba5fd-a12e-4cce-9686-0ab7bbc467f1" containerName="payload-pre-processing"
Jul 23 23:00:19.430805 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.430784    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:19.433348 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.433325    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-70930e-serving-cert\""
Jul 23 23:00:19.433487 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.433395    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 23:00:19.434280 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.434261    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 23:00:19.434404 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.434263    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vw8vx\""
Jul 23 23:00:19.440792 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.440770    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"]
Jul 23 23:00:19.519045 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.518981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lcpf\" (UniqueName: \"kubernetes.io/projected/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-kube-api-access-6lcpf\") pod \"maas-api-e2e-api-a-70930e-84bb69b547-bmdzz\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:19.519374 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.519343    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-openshift-service-ca\") pod \"maas-api-e2e-api-a-70930e-84bb69b547-bmdzz\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:19.519509 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.519400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-maas-api-tls\") pod \"maas-api-e2e-api-a-70930e-84bb69b547-bmdzz\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:19.620381 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.620351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-maas-api-tls\") pod \"maas-api-e2e-api-a-70930e-84bb69b547-bmdzz\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:19.620532 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.620432    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lcpf\" (UniqueName: \"kubernetes.io/projected/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-kube-api-access-6lcpf\") pod \"maas-api-e2e-api-a-70930e-84bb69b547-bmdzz\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:19.620532 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.620488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-openshift-service-ca\") pod \"maas-api-e2e-api-a-70930e-84bb69b547-bmdzz\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:19.620676 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:00:19.620560    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-serving-cert: secret "maas-api-e2e-api-a-70930e-serving-cert" not found
Jul 23 23:00:19.620676 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:00:19.620641    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-maas-api-tls podName:747ed5be-a34c-4bc6-aa18-7fdfddf9d753 nodeName:}" failed. No retries permitted until 2026-07-23 23:00:20.120620985 +0000 UTC m=+2274.438149781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-maas-api-tls") pod "maas-api-e2e-api-a-70930e-84bb69b547-bmdzz" (UID: "747ed5be-a34c-4bc6-aa18-7fdfddf9d753") : secret "maas-api-e2e-api-a-70930e-serving-cert" not found
Jul 23 23:00:19.621190 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.621163    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-openshift-service-ca\") pod \"maas-api-e2e-api-a-70930e-84bb69b547-bmdzz\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:19.629752 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:19.629730    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lcpf\" (UniqueName: \"kubernetes.io/projected/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-kube-api-access-6lcpf\") pod \"maas-api-e2e-api-a-70930e-84bb69b547-bmdzz\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:20.111221 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.111179    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"]
Jul 23 23:00:20.114619 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.114602    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:20.117066 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.117043    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-70930e-dockercfg-6b2sk\""
Jul 23 23:00:20.117193 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.117088    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-70930e\""
Jul 23 23:00:20.123331 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.123303    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"]
Jul 23 23:00:20.123697 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.123671    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcswp\" (UniqueName: \"kubernetes.io/projected/fed996de-9b2b-4e3e-a884-71997fde7dc3-kube-api-access-zcswp\") pod \"payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk\" (UID: \"fed996de-9b2b-4e3e-a884-71997fde7dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:20.123824 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.123738    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed996de-9b2b-4e3e-a884-71997fde7dc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk\" (UID: \"fed996de-9b2b-4e3e-a884-71997fde7dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:20.123888 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.123837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-maas-api-tls\") pod \"maas-api-e2e-api-a-70930e-84bb69b547-bmdzz\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:20.126674 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.126647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-maas-api-tls\") pod \"maas-api-e2e-api-a-70930e-84bb69b547-bmdzz\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:20.152716 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.152686    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"]
Jul 23 23:00:20.156014 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.155996    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:20.166260 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.166229    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"]
Jul 23 23:00:20.224823 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.224783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbt9\" (UniqueName: \"kubernetes.io/projected/4dcc89b1-97b7-45de-9981-2d991c78077c-kube-api-access-wdbt9\") pod \"payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f\" (UID: \"4dcc89b1-97b7-45de-9981-2d991c78077c\") " pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:20.225343 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.224848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcswp\" (UniqueName: \"kubernetes.io/projected/fed996de-9b2b-4e3e-a884-71997fde7dc3-kube-api-access-zcswp\") pod \"payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk\" (UID: \"fed996de-9b2b-4e3e-a884-71997fde7dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:20.225343 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.224877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcc89b1-97b7-45de-9981-2d991c78077c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f\" (UID: \"4dcc89b1-97b7-45de-9981-2d991c78077c\") " pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:20.225343 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.224961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed996de-9b2b-4e3e-a884-71997fde7dc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk\" (UID: \"fed996de-9b2b-4e3e-a884-71997fde7dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:20.225862 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.225837    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed996de-9b2b-4e3e-a884-71997fde7dc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk\" (UID: \"fed996de-9b2b-4e3e-a884-71997fde7dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:20.233244 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.233214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcswp\" (UniqueName: \"kubernetes.io/projected/fed996de-9b2b-4e3e-a884-71997fde7dc3-kube-api-access-zcswp\") pod \"payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk\" (UID: \"fed996de-9b2b-4e3e-a884-71997fde7dc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:20.326254 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.326209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbt9\" (UniqueName: \"kubernetes.io/projected/4dcc89b1-97b7-45de-9981-2d991c78077c-kube-api-access-wdbt9\") pod \"payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f\" (UID: \"4dcc89b1-97b7-45de-9981-2d991c78077c\") " pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:20.326416 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.326273    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcc89b1-97b7-45de-9981-2d991c78077c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f\" (UID: \"4dcc89b1-97b7-45de-9981-2d991c78077c\") " pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:20.327072 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.327054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcc89b1-97b7-45de-9981-2d991c78077c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f\" (UID: \"4dcc89b1-97b7-45de-9981-2d991c78077c\") " pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:20.333850 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.333822    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbt9\" (UniqueName: \"kubernetes.io/projected/4dcc89b1-97b7-45de-9981-2d991c78077c-kube-api-access-wdbt9\") pod \"payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f\" (UID: \"4dcc89b1-97b7-45de-9981-2d991c78077c\") " pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:20.343507 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.343482    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:20.426791 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.426753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:20.468202 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.468165    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:20.482276 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.482088    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"]
Jul 23 23:00:20.485279 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:00:20.485237    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747ed5be_a34c_4bc6_aa18_7fdfddf9d753.slice/crio-4f4eea861b14aae46c40199d223470a5bf0ddd84e4132cdb0fbc9f7bc4e258f7 WatchSource:0}: Error finding container 4f4eea861b14aae46c40199d223470a5bf0ddd84e4132cdb0fbc9f7bc4e258f7: Status 404 returned error can't find the container with id 4f4eea861b14aae46c40199d223470a5bf0ddd84e4132cdb0fbc9f7bc4e258f7
Jul 23 23:00:20.575411 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.575221    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"]
Jul 23 23:00:20.630448 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.630419    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"]
Jul 23 23:00:20.631173 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.631139    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz" event={"ID":"747ed5be-a34c-4bc6-aa18-7fdfddf9d753","Type":"ContainerStarted","Data":"4f4eea861b14aae46c40199d223470a5bf0ddd84e4132cdb0fbc9f7bc4e258f7"}
Jul 23 23:00:20.632634 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:20.632607    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk" event={"ID":"fed996de-9b2b-4e3e-a884-71997fde7dc3","Type":"ContainerStarted","Data":"7ba6cd6580072e18e4a7515cf15841d942b5ad618ec8be55c5bc7a1d9926b4a9"}
Jul 23 23:00:20.635136 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:00:20.635113    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcc89b1_97b7_45de_9981_2d991c78077c.slice/crio-c4489d53076cd52a68cb85ecea6d06937b214399508f3f3caab9b0a55caa0e7a WatchSource:0}: Error finding container c4489d53076cd52a68cb85ecea6d06937b214399508f3f3caab9b0a55caa0e7a: Status 404 returned error can't find the container with id c4489d53076cd52a68cb85ecea6d06937b214399508f3f3caab9b0a55caa0e7a
Jul 23 23:00:21.639901 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:21.639858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz" event={"ID":"747ed5be-a34c-4bc6-aa18-7fdfddf9d753","Type":"ContainerStarted","Data":"cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba"}
Jul 23 23:00:21.640388 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:21.640087    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:21.641659 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:21.641628    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f" event={"ID":"4dcc89b1-97b7-45de-9981-2d991c78077c","Type":"ContainerStarted","Data":"d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d"}
Jul 23 23:00:21.641813 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:21.641678    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f" event={"ID":"4dcc89b1-97b7-45de-9981-2d991c78077c","Type":"ContainerStarted","Data":"c4489d53076cd52a68cb85ecea6d06937b214399508f3f3caab9b0a55caa0e7a"}
Jul 23 23:00:21.643634 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:21.643605    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk" event={"ID":"fed996de-9b2b-4e3e-a884-71997fde7dc3","Type":"ContainerStarted","Data":"e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23"}
Jul 23 23:00:21.643768 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:21.643743    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:21.657375 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:21.657302    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz" podStartSLOduration=1.923219557 podStartE2EDuration="2.657289478s" totalImagesPullingTime="734.069921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:00:21.65602792 +0000 UTC m=+2275.973556986" watchObservedRunningTime="2026-07-23 23:00:21.657289478 +0000 UTC m=+2275.974818286"
Jul 23 23:00:21.671492 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:21.671438    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk" podStartSLOduration=1.671423965 podStartE2EDuration="1.671423965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:00:21.669419291 +0000 UTC m=+2275.986948100" watchObservedRunningTime="2026-07-23 23:00:21.671423965 +0000 UTC m=+2275.988952776"
Jul 23 23:00:21.686026 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:21.685979    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f" podStartSLOduration=1.685963798 podStartE2EDuration="1.685963798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:00:21.684086279 +0000 UTC m=+2276.001615101" watchObservedRunningTime="2026-07-23 23:00:21.685963798 +0000 UTC m=+2276.003492662"
Jul 23 23:00:22.649916 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:22.649869    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:27.471329 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.471278    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"]
Jul 23 23:00:27.476434 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.476407    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:27.479071 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.479045    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-857bc8-serving-cert\""
Jul 23 23:00:27.486399 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.486067    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"]
Jul 23 23:00:27.592792 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.592743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1343b946-6509-43e3-a6dd-a079e8b289d8-maas-api-tls\") pod \"maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:27.593003 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.592820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb62k\" (UniqueName: \"kubernetes.io/projected/1343b946-6509-43e3-a6dd-a079e8b289d8-kube-api-access-vb62k\") pod \"maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:27.593003 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.592853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1343b946-6509-43e3-a6dd-a079e8b289d8-openshift-service-ca\") pod \"maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:27.655323 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.655292    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:27.693747 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.693710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1343b946-6509-43e3-a6dd-a079e8b289d8-maas-api-tls\") pod \"maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:27.693937 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.693805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb62k\" (UniqueName: \"kubernetes.io/projected/1343b946-6509-43e3-a6dd-a079e8b289d8-kube-api-access-vb62k\") pod \"maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:27.693937 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.693840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1343b946-6509-43e3-a6dd-a079e8b289d8-openshift-service-ca\") pod \"maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:27.694086 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:00:27.693946    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-serving-cert: secret "maas-api-e2e-api-b-857bc8-serving-cert" not found
Jul 23 23:00:27.694086 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:00:27.694017    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1343b946-6509-43e3-a6dd-a079e8b289d8-maas-api-tls podName:1343b946-6509-43e3-a6dd-a079e8b289d8 nodeName:}" failed. No retries permitted until 2026-07-23 23:00:28.193999097 +0000 UTC m=+2282.511527903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1343b946-6509-43e3-a6dd-a079e8b289d8-maas-api-tls") pod "maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s" (UID: "1343b946-6509-43e3-a6dd-a079e8b289d8") : secret "maas-api-e2e-api-b-857bc8-serving-cert" not found
Jul 23 23:00:27.694863 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.694832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1343b946-6509-43e3-a6dd-a079e8b289d8-openshift-service-ca\") pod \"maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:27.706618 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:27.706582    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb62k\" (UniqueName: \"kubernetes.io/projected/1343b946-6509-43e3-a6dd-a079e8b289d8-kube-api-access-vb62k\") pod \"maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:28.135461 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.135425    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"]
Jul 23 23:00:28.139046 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.139024    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:00:28.141565 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.141542    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-857bc8-dockercfg-xn7tg\""
Jul 23 23:00:28.141671 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.141629    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-857bc8\""
Jul 23 23:00:28.146749 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.146724    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"]
Jul 23 23:00:28.179749 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.179712    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"]
Jul 23 23:00:28.183128 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.183108    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:00:28.192917 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.192891    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"]
Jul 23 23:00:28.198993 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.198966    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5cz\" (UniqueName: \"kubernetes.io/projected/0fea6589-112e-4b89-b32e-30905bbeb59c-kube-api-access-hw5cz\") pod \"payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg\" (UID: \"0fea6589-112e-4b89-b32e-30905bbeb59c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:00:28.199121 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.199060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1343b946-6509-43e3-a6dd-a079e8b289d8-maas-api-tls\") pod \"maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:28.199121 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.199101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fea6589-112e-4b89-b32e-30905bbeb59c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg\" (UID: \"0fea6589-112e-4b89-b32e-30905bbeb59c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:00:28.201515 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.201493    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1343b946-6509-43e3-a6dd-a079e8b289d8-maas-api-tls\") pod \"maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:28.299717 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.299675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh44c\" (UniqueName: \"kubernetes.io/projected/b7de8f53-e131-4511-9116-0e64de826c7c-kube-api-access-mh44c\") pod \"payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b\" (UID: \"b7de8f53-e131-4511-9116-0e64de826c7c\") " pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:00:28.299717 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.299715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fea6589-112e-4b89-b32e-30905bbeb59c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg\" (UID: \"0fea6589-112e-4b89-b32e-30905bbeb59c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:00:28.299964 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.299804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7de8f53-e131-4511-9116-0e64de826c7c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b\" (UID: \"b7de8f53-e131-4511-9116-0e64de826c7c\") " pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:00:28.299964 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.299876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5cz\" (UniqueName: \"kubernetes.io/projected/0fea6589-112e-4b89-b32e-30905bbeb59c-kube-api-access-hw5cz\") pod \"payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg\" (UID: \"0fea6589-112e-4b89-b32e-30905bbeb59c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:00:28.300407 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.300390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fea6589-112e-4b89-b32e-30905bbeb59c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg\" (UID: \"0fea6589-112e-4b89-b32e-30905bbeb59c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:00:28.307541 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.307518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5cz\" (UniqueName: \"kubernetes.io/projected/0fea6589-112e-4b89-b32e-30905bbeb59c-kube-api-access-hw5cz\") pod \"payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg\" (UID: \"0fea6589-112e-4b89-b32e-30905bbeb59c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:00:28.392653 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.392559    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:28.401123 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.401096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7de8f53-e131-4511-9116-0e64de826c7c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b\" (UID: \"b7de8f53-e131-4511-9116-0e64de826c7c\") " pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:00:28.401296 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.401268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh44c\" (UniqueName: \"kubernetes.io/projected/b7de8f53-e131-4511-9116-0e64de826c7c-kube-api-access-mh44c\") pod \"payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b\" (UID: \"b7de8f53-e131-4511-9116-0e64de826c7c\") " pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:00:28.401892 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.401865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7de8f53-e131-4511-9116-0e64de826c7c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b\" (UID: \"b7de8f53-e131-4511-9116-0e64de826c7c\") " pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:00:28.412956 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.412915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh44c\" (UniqueName: \"kubernetes.io/projected/b7de8f53-e131-4511-9116-0e64de826c7c-kube-api-access-mh44c\") pod \"payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b\" (UID: \"b7de8f53-e131-4511-9116-0e64de826c7c\") " pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:00:28.451545 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.451513    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:00:28.495861 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.494123    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:00:28.610436 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.610315    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"]
Jul 23 23:00:28.613889 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:00:28.613862    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fea6589_112e_4b89_b32e_30905bbeb59c.slice/crio-4eeaf012f66057ec9b602e3d0abd4d9d05b0b5fef32efd73e2ee48e52b72896a WatchSource:0}: Error finding container 4eeaf012f66057ec9b602e3d0abd4d9d05b0b5fef32efd73e2ee48e52b72896a: Status 404 returned error can't find the container with id 4eeaf012f66057ec9b602e3d0abd4d9d05b0b5fef32efd73e2ee48e52b72896a
Jul 23 23:00:28.662301 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.662277    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"]
Jul 23 23:00:28.670104 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:00:28.670077    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7de8f53_e131_4511_9116_0e64de826c7c.slice/crio-13c4b85e3d73ae746a6b992f6e2fc6c7546dc09b9bcf7a892a51a262869c459d WatchSource:0}: Error finding container 13c4b85e3d73ae746a6b992f6e2fc6c7546dc09b9bcf7a892a51a262869c459d: Status 404 returned error can't find the container with id 13c4b85e3d73ae746a6b992f6e2fc6c7546dc09b9bcf7a892a51a262869c459d
Jul 23 23:00:28.681369 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.681339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg" event={"ID":"0fea6589-112e-4b89-b32e-30905bbeb59c","Type":"ContainerStarted","Data":"4eeaf012f66057ec9b602e3d0abd4d9d05b0b5fef32efd73e2ee48e52b72896a"}
Jul 23 23:00:28.682598 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.682577    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b" event={"ID":"b7de8f53-e131-4511-9116-0e64de826c7c","Type":"ContainerStarted","Data":"13c4b85e3d73ae746a6b992f6e2fc6c7546dc09b9bcf7a892a51a262869c459d"}
Jul 23 23:00:28.766520 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:28.766493    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"]
Jul 23 23:00:28.768665 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:00:28.768639    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1343b946_6509_43e3_a6dd_a079e8b289d8.slice/crio-6d971e8400fec787ba2db61ea46781bf92d095e5a41be79845cffcee332cc2c3 WatchSource:0}: Error finding container 6d971e8400fec787ba2db61ea46781bf92d095e5a41be79845cffcee332cc2c3: Status 404 returned error can't find the container with id 6d971e8400fec787ba2db61ea46781bf92d095e5a41be79845cffcee332cc2c3
Jul 23 23:00:29.694943 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:29.694874    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b" event={"ID":"b7de8f53-e131-4511-9116-0e64de826c7c","Type":"ContainerStarted","Data":"71a0e4f3886229b3fa1dd8dc64b6bce3b88a9597d21a09e065f36b2fe14dc4ce"}
Jul 23 23:00:29.695429 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:29.695102    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:00:29.696737 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:29.696705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg" event={"ID":"0fea6589-112e-4b89-b32e-30905bbeb59c","Type":"ContainerStarted","Data":"316de81442be4e636fa4f2ac8c39329d59a808588717888ed57a16e9e2d496c5"}
Jul 23 23:00:29.696864 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:29.696762    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:00:29.698151 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:29.698124    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s" event={"ID":"1343b946-6509-43e3-a6dd-a079e8b289d8","Type":"ContainerStarted","Data":"8d7168b1e083cd04c2f3f1dd9c9bc694d41c50f218cb136e57bfe22023240fd2"}
Jul 23 23:00:29.698259 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:29.698172    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s" event={"ID":"1343b946-6509-43e3-a6dd-a079e8b289d8","Type":"ContainerStarted","Data":"6d971e8400fec787ba2db61ea46781bf92d095e5a41be79845cffcee332cc2c3"}
Jul 23 23:00:29.698259 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:29.698202    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:29.709317 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:29.709221    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b" podStartSLOduration=1.709203831 podStartE2EDuration="1.709203831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:00:29.708190568 +0000 UTC m=+2284.025719388" watchObservedRunningTime="2026-07-23 23:00:29.709203831 +0000 UTC m=+2284.026732640"
Jul 23 23:00:29.722521 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:29.722472    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg" podStartSLOduration=1.722458252 podStartE2EDuration="1.722458252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:00:29.721471753 +0000 UTC m=+2284.039000562" watchObservedRunningTime="2026-07-23 23:00:29.722458252 +0000 UTC m=+2284.039987125"
Jul 23 23:00:29.739335 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:29.739257    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s" podStartSLOduration=2.051378075 podStartE2EDuration="2.739235295s" totalImagesPullingTime="687.85722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:00:29.736002237 +0000 UTC m=+2284.053531046" watchObservedRunningTime="2026-07-23 23:00:29.739235295 +0000 UTC m=+2284.056764107"
Jul 23 23:00:32.651515 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:32.651460    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk" podUID="fed996de-9b2b-4e3e-a884-71997fde7dc3" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.62:9004: connect: connection refused"
Jul 23 23:00:33.656476 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:33.656442    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:35.709048 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:35.709015    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:00:40.706031 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:40.705989    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:00:40.706523 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:40.706063    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:00:42.650256 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:42.650224    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:55.889150 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:55.889109    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"]
Jul 23 23:00:55.889592 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:55.889336    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz" podUID="747ed5be-a34c-4bc6-aa18-7fdfddf9d753" containerName="maas-api" containerID="cri-o://cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba" gracePeriod=30
Jul 23 23:00:55.942179 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:55.942140    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"]
Jul 23 23:00:55.942409 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:55.942379    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f" podUID="4dcc89b1-97b7-45de-9981-2d991c78077c" containerName="payload-processing" containerID="cri-o://d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d" gracePeriod=30
Jul 23 23:00:55.948456 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:55.948428    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"]
Jul 23 23:00:55.948834 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:55.948805    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk" podUID="fed996de-9b2b-4e3e-a884-71997fde7dc3" containerName="payload-pre-processing" containerID="cri-o://e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23" gracePeriod=30
Jul 23 23:00:56.229076 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.229051    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:56.233223 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.233199    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:56.255217 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.255192    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:56.269995 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.269961    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcc89b1-97b7-45de-9981-2d991c78077c-plugins-config-volume\") pod \"4dcc89b1-97b7-45de-9981-2d991c78077c\" (UID: \"4dcc89b1-97b7-45de-9981-2d991c78077c\") "
Jul 23 23:00:56.270127 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.269999    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lcpf\" (UniqueName: \"kubernetes.io/projected/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-kube-api-access-6lcpf\") pod \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") "
Jul 23 23:00:56.270127 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.270029    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcswp\" (UniqueName: \"kubernetes.io/projected/fed996de-9b2b-4e3e-a884-71997fde7dc3-kube-api-access-zcswp\") pod \"fed996de-9b2b-4e3e-a884-71997fde7dc3\" (UID: \"fed996de-9b2b-4e3e-a884-71997fde7dc3\") "
Jul 23 23:00:56.270127 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.270052    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-maas-api-tls\") pod \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") "
Jul 23 23:00:56.270127 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.270068    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-openshift-service-ca\") pod \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\" (UID: \"747ed5be-a34c-4bc6-aa18-7fdfddf9d753\") "
Jul 23 23:00:56.270127 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.270088    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdbt9\" (UniqueName: \"kubernetes.io/projected/4dcc89b1-97b7-45de-9981-2d991c78077c-kube-api-access-wdbt9\") pod \"4dcc89b1-97b7-45de-9981-2d991c78077c\" (UID: \"4dcc89b1-97b7-45de-9981-2d991c78077c\") "
Jul 23 23:00:56.270127 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.270111    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed996de-9b2b-4e3e-a884-71997fde7dc3-plugins-config-volume\") pod \"fed996de-9b2b-4e3e-a884-71997fde7dc3\" (UID: \"fed996de-9b2b-4e3e-a884-71997fde7dc3\") "
Jul 23 23:00:56.270545 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.270438    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dcc89b1-97b7-45de-9981-2d991c78077c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4dcc89b1-97b7-45de-9981-2d991c78077c" (UID: "4dcc89b1-97b7-45de-9981-2d991c78077c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:00:56.270649 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.270623    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed996de-9b2b-4e3e-a884-71997fde7dc3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fed996de-9b2b-4e3e-a884-71997fde7dc3" (UID: "fed996de-9b2b-4e3e-a884-71997fde7dc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:00:56.270771 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.270750    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "747ed5be-a34c-4bc6-aa18-7fdfddf9d753" (UID: "747ed5be-a34c-4bc6-aa18-7fdfddf9d753"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:00:56.272265 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.272236    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "747ed5be-a34c-4bc6-aa18-7fdfddf9d753" (UID: "747ed5be-a34c-4bc6-aa18-7fdfddf9d753"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 23:00:56.272752 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.272734    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-kube-api-access-6lcpf" (OuterVolumeSpecName: "kube-api-access-6lcpf") pod "747ed5be-a34c-4bc6-aa18-7fdfddf9d753" (UID: "747ed5be-a34c-4bc6-aa18-7fdfddf9d753"). InnerVolumeSpecName "kube-api-access-6lcpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:00:56.272804 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.272764    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed996de-9b2b-4e3e-a884-71997fde7dc3-kube-api-access-zcswp" (OuterVolumeSpecName: "kube-api-access-zcswp") pod "fed996de-9b2b-4e3e-a884-71997fde7dc3" (UID: "fed996de-9b2b-4e3e-a884-71997fde7dc3"). InnerVolumeSpecName "kube-api-access-zcswp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:00:56.272804 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.272769    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dcc89b1-97b7-45de-9981-2d991c78077c-kube-api-access-wdbt9" (OuterVolumeSpecName: "kube-api-access-wdbt9") pod "4dcc89b1-97b7-45de-9981-2d991c78077c" (UID: "4dcc89b1-97b7-45de-9981-2d991c78077c"). InnerVolumeSpecName "kube-api-access-wdbt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:00:56.371280 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.371240    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lcpf\" (UniqueName: \"kubernetes.io/projected/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-kube-api-access-6lcpf\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:56.371280 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.371278    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcswp\" (UniqueName: \"kubernetes.io/projected/fed996de-9b2b-4e3e-a884-71997fde7dc3-kube-api-access-zcswp\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:56.371490 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.371293    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:56.371490 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.371302    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747ed5be-a34c-4bc6-aa18-7fdfddf9d753-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:56.371490 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.371312    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdbt9\" (UniqueName: \"kubernetes.io/projected/4dcc89b1-97b7-45de-9981-2d991c78077c-kube-api-access-wdbt9\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:56.371490 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.371321    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed996de-9b2b-4e3e-a884-71997fde7dc3-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:56.371490 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.371329    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcc89b1-97b7-45de-9981-2d991c78077c-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:00:56.841212 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.841175    2583 generic.go:356] "Generic (PLEG): container finished" podID="4dcc89b1-97b7-45de-9981-2d991c78077c" containerID="d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d" exitCode=0
Jul 23 23:00:56.841399 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.841230    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"
Jul 23 23:00:56.841399 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.841260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f" event={"ID":"4dcc89b1-97b7-45de-9981-2d991c78077c","Type":"ContainerDied","Data":"d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d"}
Jul 23 23:00:56.841399 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.841341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f" event={"ID":"4dcc89b1-97b7-45de-9981-2d991c78077c","Type":"ContainerDied","Data":"c4489d53076cd52a68cb85ecea6d06937b214399508f3f3caab9b0a55caa0e7a"}
Jul 23 23:00:56.841399 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.841370    2583 scope.go:117] "RemoveContainer" containerID="d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d"
Jul 23 23:00:56.842701 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.842671    2583 generic.go:356] "Generic (PLEG): container finished" podID="fed996de-9b2b-4e3e-a884-71997fde7dc3" containerID="e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23" exitCode=0
Jul 23 23:00:56.842808 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.842712    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"
Jul 23 23:00:56.842808 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.842737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk" event={"ID":"fed996de-9b2b-4e3e-a884-71997fde7dc3","Type":"ContainerDied","Data":"e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23"}
Jul 23 23:00:56.842808 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.842777    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk" event={"ID":"fed996de-9b2b-4e3e-a884-71997fde7dc3","Type":"ContainerDied","Data":"7ba6cd6580072e18e4a7515cf15841d942b5ad618ec8be55c5bc7a1d9926b4a9"}
Jul 23 23:00:56.844337 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.844311    2583 generic.go:356] "Generic (PLEG): container finished" podID="747ed5be-a34c-4bc6-aa18-7fdfddf9d753" containerID="cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba" exitCode=0
Jul 23 23:00:56.844450 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.844352    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz" event={"ID":"747ed5be-a34c-4bc6-aa18-7fdfddf9d753","Type":"ContainerDied","Data":"cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba"}
Jul 23 23:00:56.844450 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.844388    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz" event={"ID":"747ed5be-a34c-4bc6-aa18-7fdfddf9d753","Type":"ContainerDied","Data":"4f4eea861b14aae46c40199d223470a5bf0ddd84e4132cdb0fbc9f7bc4e258f7"}
Jul 23 23:00:56.844450 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.844394    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"
Jul 23 23:00:56.851458 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.851439    2583 scope.go:117] "RemoveContainer" containerID="d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d"
Jul 23 23:00:56.851741 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:00:56.851715    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d\": container with ID starting with d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d not found: ID does not exist" containerID="d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d"
Jul 23 23:00:56.851802 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.851754    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d"} err="failed to get container status \"d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d\": rpc error: code = NotFound desc = could not find container \"d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d\": container with ID starting with d29ab6b75f47526ba88b328b2fcb20ee18d66dd2a1de3462bd5e6adfdbe7b45d not found: ID does not exist"
Jul 23 23:00:56.851802 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.851775    2583 scope.go:117] "RemoveContainer" containerID="e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23"
Jul 23 23:00:56.860097 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.860078    2583 scope.go:117] "RemoveContainer" containerID="e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23"
Jul 23 23:00:56.860342 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:00:56.860324    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23\": container with ID starting with e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23 not found: ID does not exist" containerID="e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23"
Jul 23 23:00:56.860399 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.860348    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23"} err="failed to get container status \"e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23\": rpc error: code = NotFound desc = could not find container \"e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23\": container with ID starting with e5b9cedfd00d892ebfea8f4687855019d7e37fa3537eb3a3c719860a58fc2a23 not found: ID does not exist"
Jul 23 23:00:56.860399 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.860363    2583 scope.go:117] "RemoveContainer" containerID="cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba"
Jul 23 23:00:56.868518 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.868495    2583 scope.go:117] "RemoveContainer" containerID="cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba"
Jul 23 23:00:56.868774 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:00:56.868754    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba\": container with ID starting with cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba not found: ID does not exist" containerID="cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba"
Jul 23 23:00:56.868838 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.868781    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba"} err="failed to get container status \"cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba\": rpc error: code = NotFound desc = could not find container \"cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba\": container with ID starting with cf8bc86bfadde372ce9c169b1a15c88735025f859c6539c0a16f374d8f2db3ba not found: ID does not exist"
Jul 23 23:00:56.870253 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.870232    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"]
Jul 23 23:00:56.873462 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.873430    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-70930e-75cf967cfb-xhc9f"]
Jul 23 23:00:56.887000 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.886976    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"]
Jul 23 23:00:56.890774 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.890752    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70930e-84bb69b547-bmdzz"]
Jul 23 23:00:56.901749 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.901723    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"]
Jul 23 23:00:56.906139 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:56.906121    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70930e-ff9799448-kcxbk"]
Jul 23 23:00:58.228869 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:58.228833    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dcc89b1-97b7-45de-9981-2d991c78077c" path="/var/lib/kubelet/pods/4dcc89b1-97b7-45de-9981-2d991c78077c/volumes"
Jul 23 23:00:58.229259 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:58.229244    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="747ed5be-a34c-4bc6-aa18-7fdfddf9d753" path="/var/lib/kubelet/pods/747ed5be-a34c-4bc6-aa18-7fdfddf9d753/volumes"
Jul 23 23:00:58.229567 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:00:58.229555    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fed996de-9b2b-4e3e-a884-71997fde7dc3" path="/var/lib/kubelet/pods/fed996de-9b2b-4e3e-a884-71997fde7dc3/volumes"
Jul 23 23:01:07.220317 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.220280    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"]
Jul 23 23:01:07.220973 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.220717    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed996de-9b2b-4e3e-a884-71997fde7dc3" containerName="payload-pre-processing"
Jul 23 23:01:07.220973 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.220732    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed996de-9b2b-4e3e-a884-71997fde7dc3" containerName="payload-pre-processing"
Jul 23 23:01:07.220973 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.220756    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dcc89b1-97b7-45de-9981-2d991c78077c" containerName="payload-processing"
Jul 23 23:01:07.220973 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.220761    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dcc89b1-97b7-45de-9981-2d991c78077c" containerName="payload-processing"
Jul 23 23:01:07.220973 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.220769    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747ed5be-a34c-4bc6-aa18-7fdfddf9d753" containerName="maas-api"
Jul 23 23:01:07.220973 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.220776    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="747ed5be-a34c-4bc6-aa18-7fdfddf9d753" containerName="maas-api"
Jul 23 23:01:07.220973 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.220831    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dcc89b1-97b7-45de-9981-2d991c78077c" containerName="payload-processing"
Jul 23 23:01:07.220973 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.220840    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fed996de-9b2b-4e3e-a884-71997fde7dc3" containerName="payload-pre-processing"
Jul 23 23:01:07.220973 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.220847    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="747ed5be-a34c-4bc6-aa18-7fdfddf9d753" containerName="maas-api"
Jul 23 23:01:07.224258 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.224230    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"
Jul 23 23:01:07.226624 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.226604    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6cq4\""
Jul 23 23:01:07.230606 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.230581    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"]
Jul 23 23:01:07.261761 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.261729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6f6d4a-4b2d-4088-a008-e8dabefe94cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c\" (UID: \"6c6f6d4a-4b2d-4088-a008-e8dabefe94cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"
Jul 23 23:01:07.362683 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.362647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6f6d4a-4b2d-4088-a008-e8dabefe94cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c\" (UID: \"6c6f6d4a-4b2d-4088-a008-e8dabefe94cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"
Jul 23 23:01:07.363308 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.363290    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6f6d4a-4b2d-4088-a008-e8dabefe94cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c\" (UID: \"6c6f6d4a-4b2d-4088-a008-e8dabefe94cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"
Jul 23 23:01:07.537730 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.537644    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"
Jul 23 23:01:07.680346 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.680314    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"]
Jul 23 23:01:07.682604 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:01:07.682581    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6f6d4a_4b2d_4088_a008_e8dabefe94cc.slice/crio-4845e420493d39deadf73a91d25856e9230a6e91b0bbd9638d41a641e0c6aef2 WatchSource:0}: Error finding container 4845e420493d39deadf73a91d25856e9230a6e91b0bbd9638d41a641e0c6aef2: Status 404 returned error can't find the container with id 4845e420493d39deadf73a91d25856e9230a6e91b0bbd9638d41a641e0c6aef2
Jul 23 23:01:07.906135 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:07.906100    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c" event={"ID":"6c6f6d4a-4b2d-4088-a008-e8dabefe94cc","Type":"ContainerStarted","Data":"4845e420493d39deadf73a91d25856e9230a6e91b0bbd9638d41a641e0c6aef2"}
Jul 23 23:01:08.912791 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:08.912756    2583 generic.go:356] "Generic (PLEG): container finished" podID="6c6f6d4a-4b2d-4088-a008-e8dabefe94cc" containerID="78d66a76c6fb95547621a3ad174ce41d8071ec3e965d0ab8949024ff0b21c57b" exitCode=0
Jul 23 23:01:08.913232 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:08.912811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c" event={"ID":"6c6f6d4a-4b2d-4088-a008-e8dabefe94cc","Type":"ContainerDied","Data":"78d66a76c6fb95547621a3ad174ce41d8071ec3e965d0ab8949024ff0b21c57b"}
Jul 23 23:01:10.053077 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:10.053050    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"
Jul 23 23:01:10.087004 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:10.086969    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6f6d4a-4b2d-4088-a008-e8dabefe94cc-maas-api-service-ca\") pod \"6c6f6d4a-4b2d-4088-a008-e8dabefe94cc\" (UID: \"6c6f6d4a-4b2d-4088-a008-e8dabefe94cc\") "
Jul 23 23:01:10.087339 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:10.087315    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6f6d4a-4b2d-4088-a008-e8dabefe94cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c6f6d4a-4b2d-4088-a008-e8dabefe94cc" (UID: "6c6f6d4a-4b2d-4088-a008-e8dabefe94cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:01:10.188490 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:10.188408    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6f6d4a-4b2d-4088-a008-e8dabefe94cc-maas-api-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:01:10.923254 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:10.923223    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"
Jul 23 23:01:10.923433 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:10.923222    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c" event={"ID":"6c6f6d4a-4b2d-4088-a008-e8dabefe94cc","Type":"ContainerDied","Data":"4845e420493d39deadf73a91d25856e9230a6e91b0bbd9638d41a641e0c6aef2"}
Jul 23 23:01:10.923433 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:10.923349    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4845e420493d39deadf73a91d25856e9230a6e91b0bbd9638d41a641e0c6aef2"
Jul 23 23:01:22.772249 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.772171    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"]
Jul 23 23:01:22.772647 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.772398    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s" podUID="1343b946-6509-43e3-a6dd-a079e8b289d8" containerName="maas-api" containerID="cri-o://8d7168b1e083cd04c2f3f1dd9c9bc694d41c50f218cb136e57bfe22023240fd2" gracePeriod=30
Jul 23 23:01:22.821596 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.821559    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"]
Jul 23 23:01:22.821871 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.821812    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b" podUID="b7de8f53-e131-4511-9116-0e64de826c7c" containerName="payload-processing" containerID="cri-o://71a0e4f3886229b3fa1dd8dc64b6bce3b88a9597d21a09e065f36b2fe14dc4ce" gracePeriod=30
Jul 23 23:01:22.830741 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.830691    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"]
Jul 23 23:01:22.830973 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.830916    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg" podUID="0fea6589-112e-4b89-b32e-30905bbeb59c" containerName="payload-pre-processing" containerID="cri-o://316de81442be4e636fa4f2ac8c39329d59a808588717888ed57a16e9e2d496c5" gracePeriod=30
Jul 23 23:01:22.984583 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.984515    2583 generic.go:356] "Generic (PLEG): container finished" podID="b7de8f53-e131-4511-9116-0e64de826c7c" containerID="71a0e4f3886229b3fa1dd8dc64b6bce3b88a9597d21a09e065f36b2fe14dc4ce" exitCode=0
Jul 23 23:01:22.984774 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.984677    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b" event={"ID":"b7de8f53-e131-4511-9116-0e64de826c7c","Type":"ContainerDied","Data":"71a0e4f3886229b3fa1dd8dc64b6bce3b88a9597d21a09e065f36b2fe14dc4ce"}
Jul 23 23:01:22.986456 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.986402    2583 generic.go:356] "Generic (PLEG): container finished" podID="0fea6589-112e-4b89-b32e-30905bbeb59c" containerID="316de81442be4e636fa4f2ac8c39329d59a808588717888ed57a16e9e2d496c5" exitCode=0
Jul 23 23:01:22.986591 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.986506    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg" event={"ID":"0fea6589-112e-4b89-b32e-30905bbeb59c","Type":"ContainerDied","Data":"316de81442be4e636fa4f2ac8c39329d59a808588717888ed57a16e9e2d496c5"}
Jul 23 23:01:22.988645 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.988619    2583 generic.go:356] "Generic (PLEG): container finished" podID="1343b946-6509-43e3-a6dd-a079e8b289d8" containerID="8d7168b1e083cd04c2f3f1dd9c9bc694d41c50f218cb136e57bfe22023240fd2" exitCode=0
Jul 23 23:01:22.988744 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:22.988664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s" event={"ID":"1343b946-6509-43e3-a6dd-a079e8b289d8","Type":"ContainerDied","Data":"8d7168b1e083cd04c2f3f1dd9c9bc694d41c50f218cb136e57bfe22023240fd2"}
Jul 23 23:01:23.103884 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.103855    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:01:23.125358 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.125328    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:01:23.129169 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.129141    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:01:23.199989 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.199956    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1343b946-6509-43e3-a6dd-a079e8b289d8-maas-api-tls\") pod \"1343b946-6509-43e3-a6dd-a079e8b289d8\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") "
Jul 23 23:01:23.200150 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.200021    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh44c\" (UniqueName: \"kubernetes.io/projected/b7de8f53-e131-4511-9116-0e64de826c7c-kube-api-access-mh44c\") pod \"b7de8f53-e131-4511-9116-0e64de826c7c\" (UID: \"b7de8f53-e131-4511-9116-0e64de826c7c\") "
Jul 23 23:01:23.200150 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.200057    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fea6589-112e-4b89-b32e-30905bbeb59c-plugins-config-volume\") pod \"0fea6589-112e-4b89-b32e-30905bbeb59c\" (UID: \"0fea6589-112e-4b89-b32e-30905bbeb59c\") "
Jul 23 23:01:23.200150 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.200080    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7de8f53-e131-4511-9116-0e64de826c7c-plugins-config-volume\") pod \"b7de8f53-e131-4511-9116-0e64de826c7c\" (UID: \"b7de8f53-e131-4511-9116-0e64de826c7c\") "
Jul 23 23:01:23.200284 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.200150    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb62k\" (UniqueName: \"kubernetes.io/projected/1343b946-6509-43e3-a6dd-a079e8b289d8-kube-api-access-vb62k\") pod \"1343b946-6509-43e3-a6dd-a079e8b289d8\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") "
Jul 23 23:01:23.200284 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.200185    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1343b946-6509-43e3-a6dd-a079e8b289d8-openshift-service-ca\") pod \"1343b946-6509-43e3-a6dd-a079e8b289d8\" (UID: \"1343b946-6509-43e3-a6dd-a079e8b289d8\") "
Jul 23 23:01:23.200284 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.200213    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw5cz\" (UniqueName: \"kubernetes.io/projected/0fea6589-112e-4b89-b32e-30905bbeb59c-kube-api-access-hw5cz\") pod \"0fea6589-112e-4b89-b32e-30905bbeb59c\" (UID: \"0fea6589-112e-4b89-b32e-30905bbeb59c\") "
Jul 23 23:01:23.200533 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.200488    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fea6589-112e-4b89-b32e-30905bbeb59c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0fea6589-112e-4b89-b32e-30905bbeb59c" (UID: "0fea6589-112e-4b89-b32e-30905bbeb59c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:01:23.200665 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.200533    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7de8f53-e131-4511-9116-0e64de826c7c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7de8f53-e131-4511-9116-0e64de826c7c" (UID: "b7de8f53-e131-4511-9116-0e64de826c7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:01:23.200665 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.200613    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1343b946-6509-43e3-a6dd-a079e8b289d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1343b946-6509-43e3-a6dd-a079e8b289d8" (UID: "1343b946-6509-43e3-a6dd-a079e8b289d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:01:23.202256 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.202229    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1343b946-6509-43e3-a6dd-a079e8b289d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1343b946-6509-43e3-a6dd-a079e8b289d8" (UID: "1343b946-6509-43e3-a6dd-a079e8b289d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 23:01:23.202358 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.202277    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7de8f53-e131-4511-9116-0e64de826c7c-kube-api-access-mh44c" (OuterVolumeSpecName: "kube-api-access-mh44c") pod "b7de8f53-e131-4511-9116-0e64de826c7c" (UID: "b7de8f53-e131-4511-9116-0e64de826c7c"). InnerVolumeSpecName "kube-api-access-mh44c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:01:23.202358 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.202325    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1343b946-6509-43e3-a6dd-a079e8b289d8-kube-api-access-vb62k" (OuterVolumeSpecName: "kube-api-access-vb62k") pod "1343b946-6509-43e3-a6dd-a079e8b289d8" (UID: "1343b946-6509-43e3-a6dd-a079e8b289d8"). InnerVolumeSpecName "kube-api-access-vb62k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:01:23.202358 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.202346    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fea6589-112e-4b89-b32e-30905bbeb59c-kube-api-access-hw5cz" (OuterVolumeSpecName: "kube-api-access-hw5cz") pod "0fea6589-112e-4b89-b32e-30905bbeb59c" (UID: "0fea6589-112e-4b89-b32e-30905bbeb59c"). InnerVolumeSpecName "kube-api-access-hw5cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:01:23.301298 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.301256    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1343b946-6509-43e3-a6dd-a079e8b289d8-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:01:23.301298 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.301294    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh44c\" (UniqueName: \"kubernetes.io/projected/b7de8f53-e131-4511-9116-0e64de826c7c-kube-api-access-mh44c\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:01:23.301508 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.301308    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fea6589-112e-4b89-b32e-30905bbeb59c-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:01:23.301508 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.301322    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7de8f53-e131-4511-9116-0e64de826c7c-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:01:23.301508 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.301333    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb62k\" (UniqueName: \"kubernetes.io/projected/1343b946-6509-43e3-a6dd-a079e8b289d8-kube-api-access-vb62k\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:01:23.301508 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.301345    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1343b946-6509-43e3-a6dd-a079e8b289d8-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:01:23.301508 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.301375    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw5cz\" (UniqueName: \"kubernetes.io/projected/0fea6589-112e-4b89-b32e-30905bbeb59c-kube-api-access-hw5cz\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:01:23.994520 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.994488    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"
Jul 23 23:01:23.994995 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.994519    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b" event={"ID":"b7de8f53-e131-4511-9116-0e64de826c7c","Type":"ContainerDied","Data":"13c4b85e3d73ae746a6b992f6e2fc6c7546dc09b9bcf7a892a51a262869c459d"}
Jul 23 23:01:23.994995 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.994580    2583 scope.go:117] "RemoveContainer" containerID="71a0e4f3886229b3fa1dd8dc64b6bce3b88a9597d21a09e065f36b2fe14dc4ce"
Jul 23 23:01:23.995985 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.995956    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg" event={"ID":"0fea6589-112e-4b89-b32e-30905bbeb59c","Type":"ContainerDied","Data":"4eeaf012f66057ec9b602e3d0abd4d9d05b0b5fef32efd73e2ee48e52b72896a"}
Jul 23 23:01:23.996147 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.995966    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"
Jul 23 23:01:23.997448 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.997425    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s" event={"ID":"1343b946-6509-43e3-a6dd-a079e8b289d8","Type":"ContainerDied","Data":"6d971e8400fec787ba2db61ea46781bf92d095e5a41be79845cffcee332cc2c3"}
Jul 23 23:01:23.997567 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:23.997440    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"
Jul 23 23:01:24.004690 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:24.004669    2583 scope.go:117] "RemoveContainer" containerID="316de81442be4e636fa4f2ac8c39329d59a808588717888ed57a16e9e2d496c5"
Jul 23 23:01:24.012751 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:24.012736    2583 scope.go:117] "RemoveContainer" containerID="8d7168b1e083cd04c2f3f1dd9c9bc694d41c50f218cb136e57bfe22023240fd2"
Jul 23 23:01:24.020892 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:24.020874    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"]
Jul 23 23:01:24.022510 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:24.022491    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-857bc8-56d9d88cc6-6kt9b"]
Jul 23 23:01:24.032434 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:24.032414    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"]
Jul 23 23:01:24.034817 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:24.034797    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-857bc8-55ffc887f7-mkj6s"]
Jul 23 23:01:24.044615 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:24.044563    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"]
Jul 23 23:01:24.047801 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:24.047781    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-857bc8-5979d864dd-4qmdg"]
Jul 23 23:01:24.228506 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:24.228476    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fea6589-112e-4b89-b32e-30905bbeb59c" path="/var/lib/kubelet/pods/0fea6589-112e-4b89-b32e-30905bbeb59c/volumes"
Jul 23 23:01:24.228854 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:24.228841    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1343b946-6509-43e3-a6dd-a079e8b289d8" path="/var/lib/kubelet/pods/1343b946-6509-43e3-a6dd-a079e8b289d8/volumes"
Jul 23 23:01:24.229191 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:24.229179    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7de8f53-e131-4511-9116-0e64de826c7c" path="/var/lib/kubelet/pods/b7de8f53-e131-4511-9116-0e64de826c7c/volumes"
Jul 23 23:01:26.459684 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:26.459648    2583 scope.go:117] "RemoveContainer" containerID="a2c16972da19aa697ff20df064a3280b8e3bbc5cf319af5bd3b94a10757184f0"
Jul 23 23:01:27.292242 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:27.292202    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"]
Jul 23 23:01:27.294100 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:27.294076    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-857bc8-hnq6c"]
Jul 23 23:01:28.228102 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:28.228070    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6f6d4a-4b2d-4088-a008-e8dabefe94cc" path="/var/lib/kubelet/pods/6c6f6d4a-4b2d-4088-a008-e8dabefe94cc/volumes"
Jul 23 23:01:37.891870 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.891832    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"]
Jul 23 23:01:37.892389 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892345    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1343b946-6509-43e3-a6dd-a079e8b289d8" containerName="maas-api"
Jul 23 23:01:37.892389 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892368    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1343b946-6509-43e3-a6dd-a079e8b289d8" containerName="maas-api"
Jul 23 23:01:37.892389 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892390    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6f6d4a-4b2d-4088-a008-e8dabefe94cc" containerName="revoke-keys"
Jul 23 23:01:37.892563 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892399    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6f6d4a-4b2d-4088-a008-e8dabefe94cc" containerName="revoke-keys"
Jul 23 23:01:37.892563 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892417    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7de8f53-e131-4511-9116-0e64de826c7c" containerName="payload-processing"
Jul 23 23:01:37.892563 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892427    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7de8f53-e131-4511-9116-0e64de826c7c" containerName="payload-processing"
Jul 23 23:01:37.892563 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892443    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fea6589-112e-4b89-b32e-30905bbeb59c" containerName="payload-pre-processing"
Jul 23 23:01:37.892563 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892452    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fea6589-112e-4b89-b32e-30905bbeb59c" containerName="payload-pre-processing"
Jul 23 23:01:37.892563 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892531    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1343b946-6509-43e3-a6dd-a079e8b289d8" containerName="maas-api"
Jul 23 23:01:37.892563 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892548    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7de8f53-e131-4511-9116-0e64de826c7c" containerName="payload-processing"
Jul 23 23:01:37.892563 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892559    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6f6d4a-4b2d-4088-a008-e8dabefe94cc" containerName="revoke-keys"
Jul 23 23:01:37.893003 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.892569    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fea6589-112e-4b89-b32e-30905bbeb59c" containerName="payload-pre-processing"
Jul 23 23:01:37.895560 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.895541    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:37.898174 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.898148    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 23:01:37.898295 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.898219    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 23:01:37.898368 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.898148    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-fd1be5-serving-cert\""
Jul 23 23:01:37.899129 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.899105    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vw8vx\""
Jul 23 23:01:37.904649 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:37.904589    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"]
Jul 23 23:01:38.022102 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.022066    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc09e64a-9854-462b-829e-a0a1e10fad2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:38.022354 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.022111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdt7n\" (UniqueName: \"kubernetes.io/projected/cc09e64a-9854-462b-829e-a0a1e10fad2d-kube-api-access-bdt7n\") pod \"maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:38.022354 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.022184    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc09e64a-9854-462b-829e-a0a1e10fad2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:38.123421 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.123389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc09e64a-9854-462b-829e-a0a1e10fad2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:38.123605 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.123445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc09e64a-9854-462b-829e-a0a1e10fad2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:38.123605 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.123478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdt7n\" (UniqueName: \"kubernetes.io/projected/cc09e64a-9854-462b-829e-a0a1e10fad2d-kube-api-access-bdt7n\") pod \"maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:38.123605 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:01:38.123545    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-serving-cert: secret "maas-api-e2e-shared-a-fd1be5-serving-cert" not found
Jul 23 23:01:38.123718 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:01:38.123613    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc09e64a-9854-462b-829e-a0a1e10fad2d-maas-api-tls podName:cc09e64a-9854-462b-829e-a0a1e10fad2d nodeName:}" failed. No retries permitted until 2026-07-23 23:01:38.623596795 +0000 UTC m=+2352.941125582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc09e64a-9854-462b-829e-a0a1e10fad2d-maas-api-tls") pod "maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9" (UID: "cc09e64a-9854-462b-829e-a0a1e10fad2d") : secret "maas-api-e2e-shared-a-fd1be5-serving-cert" not found
Jul 23 23:01:38.124165 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.124147    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc09e64a-9854-462b-829e-a0a1e10fad2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:38.132159 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.132133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdt7n\" (UniqueName: \"kubernetes.io/projected/cc09e64a-9854-462b-829e-a0a1e10fad2d-kube-api-access-bdt7n\") pod \"maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:38.588208 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.588170    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"]
Jul 23 23:01:38.591645 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.591623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:01:38.594179 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.594155    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-fd1be5\""
Jul 23 23:01:38.594318 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.594196    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-fd1be5-dockercfg-k9cvt\""
Jul 23 23:01:38.601755 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.601726    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"]
Jul 23 23:01:38.623115 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.623078    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"]
Jul 23 23:01:38.626398 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.626376    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:01:38.628866 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.628846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc09e64a-9854-462b-829e-a0a1e10fad2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:38.631225 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.631197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc09e64a-9854-462b-829e-a0a1e10fad2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:38.636452 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.636429    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"]
Jul 23 23:01:38.730223 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.730171    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxbd\" (UniqueName: \"kubernetes.io/projected/6545665c-ab1a-4392-b5bd-59b5430499c9-kube-api-access-fxxbd\") pod \"payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d\" (UID: \"6545665c-ab1a-4392-b5bd-59b5430499c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:01:38.730425 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.730250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d93580-cc40-402e-9c16-ba19591d8cec-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9\" (UID: \"90d93580-cc40-402e-9c16-ba19591d8cec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:01:38.730425 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.730329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmmvs\" (UniqueName: \"kubernetes.io/projected/90d93580-cc40-402e-9c16-ba19591d8cec-kube-api-access-lmmvs\") pod \"payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9\" (UID: \"90d93580-cc40-402e-9c16-ba19591d8cec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:01:38.730544 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.730466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6545665c-ab1a-4392-b5bd-59b5430499c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d\" (UID: \"6545665c-ab1a-4392-b5bd-59b5430499c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:01:38.807246 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.807208    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:38.831637 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.831605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d93580-cc40-402e-9c16-ba19591d8cec-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9\" (UID: \"90d93580-cc40-402e-9c16-ba19591d8cec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:01:38.831769 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.831644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmmvs\" (UniqueName: \"kubernetes.io/projected/90d93580-cc40-402e-9c16-ba19591d8cec-kube-api-access-lmmvs\") pod \"payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9\" (UID: \"90d93580-cc40-402e-9c16-ba19591d8cec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:01:38.831769 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.831710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6545665c-ab1a-4392-b5bd-59b5430499c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d\" (UID: \"6545665c-ab1a-4392-b5bd-59b5430499c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:01:38.831769 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.831748    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxbd\" (UniqueName: \"kubernetes.io/projected/6545665c-ab1a-4392-b5bd-59b5430499c9-kube-api-access-fxxbd\") pod \"payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d\" (UID: \"6545665c-ab1a-4392-b5bd-59b5430499c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:01:38.832436 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.832413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6545665c-ab1a-4392-b5bd-59b5430499c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d\" (UID: \"6545665c-ab1a-4392-b5bd-59b5430499c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:01:38.832562 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.832527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d93580-cc40-402e-9c16-ba19591d8cec-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9\" (UID: \"90d93580-cc40-402e-9c16-ba19591d8cec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:01:38.839761 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.839699    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmmvs\" (UniqueName: \"kubernetes.io/projected/90d93580-cc40-402e-9c16-ba19591d8cec-kube-api-access-lmmvs\") pod \"payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9\" (UID: \"90d93580-cc40-402e-9c16-ba19591d8cec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:01:38.839857 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.839824    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxbd\" (UniqueName: \"kubernetes.io/projected/6545665c-ab1a-4392-b5bd-59b5430499c9-kube-api-access-fxxbd\") pod \"payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d\" (UID: \"6545665c-ab1a-4392-b5bd-59b5430499c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:01:38.906260 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.905962    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:01:38.955643 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.955596    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:01:38.958752 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:38.958722    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"]
Jul 23 23:01:38.960145 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:01:38.960116    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc09e64a_9854_462b_829e_a0a1e10fad2d.slice/crio-5cdc35f4c57e0accd6fc1a8d3ca29838e79f82f8a4275a50fa6a3d24684c41f4 WatchSource:0}: Error finding container 5cdc35f4c57e0accd6fc1a8d3ca29838e79f82f8a4275a50fa6a3d24684c41f4: Status 404 returned error can't find the container with id 5cdc35f4c57e0accd6fc1a8d3ca29838e79f82f8a4275a50fa6a3d24684c41f4
Jul 23 23:01:39.067531 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:39.067435    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"]
Jul 23 23:01:39.068713 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:01:39.068673    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6545665c_ab1a_4392_b5bd_59b5430499c9.slice/crio-a0b04cc7ea5533fc5024984dac3ba00a7aea6ff147781f3a67f8be461a10c983 WatchSource:0}: Error finding container a0b04cc7ea5533fc5024984dac3ba00a7aea6ff147781f3a67f8be461a10c983: Status 404 returned error can't find the container with id a0b04cc7ea5533fc5024984dac3ba00a7aea6ff147781f3a67f8be461a10c983
Jul 23 23:01:39.073646 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:39.073612    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9" event={"ID":"cc09e64a-9854-462b-829e-a0a1e10fad2d","Type":"ContainerStarted","Data":"5cdc35f4c57e0accd6fc1a8d3ca29838e79f82f8a4275a50fa6a3d24684c41f4"}
Jul 23 23:01:39.120415 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:01:39.120369    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d93580_cc40_402e_9c16_ba19591d8cec.slice/crio-41035b40abb1517ee26c4c32a433c982f9b1183d5a50b5f6a279f31fcbf05e4a WatchSource:0}: Error finding container 41035b40abb1517ee26c4c32a433c982f9b1183d5a50b5f6a279f31fcbf05e4a: Status 404 returned error can't find the container with id 41035b40abb1517ee26c4c32a433c982f9b1183d5a50b5f6a279f31fcbf05e4a
Jul 23 23:01:39.126287 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:39.126255    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"]
Jul 23 23:01:40.080330 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:40.080294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9" event={"ID":"cc09e64a-9854-462b-829e-a0a1e10fad2d","Type":"ContainerStarted","Data":"a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5"}
Jul 23 23:01:40.080837 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:40.080392    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:40.081864 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:40.081842    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9" event={"ID":"90d93580-cc40-402e-9c16-ba19591d8cec","Type":"ContainerStarted","Data":"2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a"}
Jul 23 23:01:40.081998 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:40.081895    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:01:40.081998 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:40.081918    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9" event={"ID":"90d93580-cc40-402e-9c16-ba19591d8cec","Type":"ContainerStarted","Data":"41035b40abb1517ee26c4c32a433c982f9b1183d5a50b5f6a279f31fcbf05e4a"}
Jul 23 23:01:40.083260 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:40.083240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d" event={"ID":"6545665c-ab1a-4392-b5bd-59b5430499c9","Type":"ContainerStarted","Data":"e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b"}
Jul 23 23:01:40.083354 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:40.083280    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d" event={"ID":"6545665c-ab1a-4392-b5bd-59b5430499c9","Type":"ContainerStarted","Data":"a0b04cc7ea5533fc5024984dac3ba00a7aea6ff147781f3a67f8be461a10c983"}
Jul 23 23:01:40.083404 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:40.083364    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:01:40.098470 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:40.098416    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9" podStartSLOduration=2.599934726 podStartE2EDuration="3.098401002s" totalImagesPullingTime="498.466276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:01:40.096051243 +0000 UTC m=+2354.413580088" watchObservedRunningTime="2026-07-23 23:01:40.098401002 +0000 UTC m=+2354.415929811"
Jul 23 23:01:40.111589 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:40.111539    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d" podStartSLOduration=2.111524557 podStartE2EDuration="2.111524557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:01:40.110308668 +0000 UTC m=+2354.427837476" watchObservedRunningTime="2026-07-23 23:01:40.111524557 +0000 UTC m=+2354.429053366"
Jul 23 23:01:40.126287 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:40.126242    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9" podStartSLOduration=2.126228254 podStartE2EDuration="2.126228254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:01:40.125395468 +0000 UTC m=+2354.442924278" watchObservedRunningTime="2026-07-23 23:01:40.126228254 +0000 UTC m=+2354.443757088"
Jul 23 23:01:45.888166 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:45.888132    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"]
Jul 23 23:01:45.892895 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:45.892868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:45.895482 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:45.895456    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-159535-serving-cert\""
Jul 23 23:01:45.903170 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:45.903145    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"]
Jul 23 23:01:45.998639 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:45.998603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae922a-1907-47f0-b8e9-89f77e76856e-maas-api-tls\") pod \"maas-api-e2e-shared-b-159535-59f995b595-kc28l\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:45.998799 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:45.998645    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j585c\" (UniqueName: \"kubernetes.io/projected/a4ae922a-1907-47f0-b8e9-89f77e76856e-kube-api-access-j585c\") pod \"maas-api-e2e-shared-b-159535-59f995b595-kc28l\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:45.998799 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:45.998671    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ae922a-1907-47f0-b8e9-89f77e76856e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-159535-59f995b595-kc28l\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:46.097144 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.097114    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:01:46.099990 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.099963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae922a-1907-47f0-b8e9-89f77e76856e-maas-api-tls\") pod \"maas-api-e2e-shared-b-159535-59f995b595-kc28l\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:46.100073 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.100011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j585c\" (UniqueName: \"kubernetes.io/projected/a4ae922a-1907-47f0-b8e9-89f77e76856e-kube-api-access-j585c\") pod \"maas-api-e2e-shared-b-159535-59f995b595-kc28l\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:46.100073 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.100064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ae922a-1907-47f0-b8e9-89f77e76856e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-159535-59f995b595-kc28l\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:46.100169 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:01:46.100093    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-serving-cert: secret "maas-api-e2e-shared-b-159535-serving-cert" not found
Jul 23 23:01:46.100169 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:01:46.100147    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ae922a-1907-47f0-b8e9-89f77e76856e-maas-api-tls podName:a4ae922a-1907-47f0-b8e9-89f77e76856e nodeName:}" failed. No retries permitted until 2026-07-23 23:01:46.600133924 +0000 UTC m=+2360.917662715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4ae922a-1907-47f0-b8e9-89f77e76856e-maas-api-tls") pod "maas-api-e2e-shared-b-159535-59f995b595-kc28l" (UID: "a4ae922a-1907-47f0-b8e9-89f77e76856e") : secret "maas-api-e2e-shared-b-159535-serving-cert" not found
Jul 23 23:01:46.100792 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.100772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ae922a-1907-47f0-b8e9-89f77e76856e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-159535-59f995b595-kc28l\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:46.112884 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.112856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j585c\" (UniqueName: \"kubernetes.io/projected/a4ae922a-1907-47f0-b8e9-89f77e76856e-kube-api-access-j585c\") pod \"maas-api-e2e-shared-b-159535-59f995b595-kc28l\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:46.576328 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.576296    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"]
Jul 23 23:01:46.579765 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.579748    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:01:46.582040 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.582017    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-159535\""
Jul 23 23:01:46.582190 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.582104    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-159535-dockercfg-nf85p\""
Jul 23 23:01:46.587109 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.587029    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"]
Jul 23 23:01:46.603444 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.603418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae922a-1907-47f0-b8e9-89f77e76856e-maas-api-tls\") pod \"maas-api-e2e-shared-b-159535-59f995b595-kc28l\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:46.605791 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.605764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae922a-1907-47f0-b8e9-89f77e76856e-maas-api-tls\") pod \"maas-api-e2e-shared-b-159535-59f995b595-kc28l\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:46.617722 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.617699    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"]
Jul 23 23:01:46.621112 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.621097    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:01:46.630438 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.630414    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"]
Jul 23 23:01:46.704088 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.704054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmkjf\" (UniqueName: \"kubernetes.io/projected/e0751b7d-9666-4640-8352-a4d72c618ca1-kube-api-access-gmkjf\") pod \"payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5\" (UID: \"e0751b7d-9666-4640-8352-a4d72c618ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:01:46.704253 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.704189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0751b7d-9666-4640-8352-a4d72c618ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5\" (UID: \"e0751b7d-9666-4640-8352-a4d72c618ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:01:46.804641 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.804603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0751b7d-9666-4640-8352-a4d72c618ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5\" (UID: \"e0751b7d-9666-4640-8352-a4d72c618ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:01:46.804844 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.804663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhrvb\" (UniqueName: \"kubernetes.io/projected/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-kube-api-access-xhrvb\") pod \"payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt\" (UID: \"b8914fd5-6a57-4ca3-88b9-b7d30837eff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:01:46.804844 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.804697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt\" (UID: \"b8914fd5-6a57-4ca3-88b9-b7d30837eff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:01:46.804844 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.804742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmkjf\" (UniqueName: \"kubernetes.io/projected/e0751b7d-9666-4640-8352-a4d72c618ca1-kube-api-access-gmkjf\") pod \"payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5\" (UID: \"e0751b7d-9666-4640-8352-a4d72c618ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:01:46.805459 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.805434    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0751b7d-9666-4640-8352-a4d72c618ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5\" (UID: \"e0751b7d-9666-4640-8352-a4d72c618ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:01:46.806595 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.806577    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:46.812482 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.812462    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmkjf\" (UniqueName: \"kubernetes.io/projected/e0751b7d-9666-4640-8352-a4d72c618ca1-kube-api-access-gmkjf\") pod \"payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5\" (UID: \"e0751b7d-9666-4640-8352-a4d72c618ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:01:46.893029 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.892986    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:01:46.905434 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.905402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhrvb\" (UniqueName: \"kubernetes.io/projected/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-kube-api-access-xhrvb\") pod \"payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt\" (UID: \"b8914fd5-6a57-4ca3-88b9-b7d30837eff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:01:46.905573 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.905465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt\" (UID: \"b8914fd5-6a57-4ca3-88b9-b7d30837eff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:01:46.906596 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.906396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt\" (UID: \"b8914fd5-6a57-4ca3-88b9-b7d30837eff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:01:46.918291 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.918265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhrvb\" (UniqueName: \"kubernetes.io/projected/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-kube-api-access-xhrvb\") pod \"payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt\" (UID: \"b8914fd5-6a57-4ca3-88b9-b7d30837eff2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:01:46.935129 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.935078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:01:46.955912 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:46.955884    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"]
Jul 23 23:01:46.958632 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:01:46.958565    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ae922a_1907_47f0_b8e9_89f77e76856e.slice/crio-684e66b53f4541b1c4e96498d44f678b8fa9fadf1b0226e43a7c89ecf1d4ec1a WatchSource:0}: Error finding container 684e66b53f4541b1c4e96498d44f678b8fa9fadf1b0226e43a7c89ecf1d4ec1a: Status 404 returned error can't find the container with id 684e66b53f4541b1c4e96498d44f678b8fa9fadf1b0226e43a7c89ecf1d4ec1a
Jul 23 23:01:47.083467 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:47.083423    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"]
Jul 23 23:01:47.086113 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:01:47.086083    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0751b7d_9666_4640_8352_a4d72c618ca1.slice/crio-c64aa4e086f3cd6b67153de8671a58465fe73b2507a35bc1b2a9340be98a1523 WatchSource:0}: Error finding container c64aa4e086f3cd6b67153de8671a58465fe73b2507a35bc1b2a9340be98a1523: Status 404 returned error can't find the container with id c64aa4e086f3cd6b67153de8671a58465fe73b2507a35bc1b2a9340be98a1523
Jul 23 23:01:47.108593 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:47.108566    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"]
Jul 23 23:01:47.116119 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:01:47.116087    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8914fd5_6a57_4ca3_88b9_b7d30837eff2.slice/crio-63f56a2909dc329201343106fbf71c7c498cf3b97b073643a0adc8b646c0b988 WatchSource:0}: Error finding container 63f56a2909dc329201343106fbf71c7c498cf3b97b073643a0adc8b646c0b988: Status 404 returned error can't find the container with id 63f56a2909dc329201343106fbf71c7c498cf3b97b073643a0adc8b646c0b988
Jul 23 23:01:47.127951 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:47.127896    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt" event={"ID":"b8914fd5-6a57-4ca3-88b9-b7d30837eff2","Type":"ContainerStarted","Data":"63f56a2909dc329201343106fbf71c7c498cf3b97b073643a0adc8b646c0b988"}
Jul 23 23:01:47.129158 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:47.129123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l" event={"ID":"a4ae922a-1907-47f0-b8e9-89f77e76856e","Type":"ContainerStarted","Data":"684e66b53f4541b1c4e96498d44f678b8fa9fadf1b0226e43a7c89ecf1d4ec1a"}
Jul 23 23:01:47.130475 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:47.130446    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5" event={"ID":"e0751b7d-9666-4640-8352-a4d72c618ca1","Type":"ContainerStarted","Data":"c64aa4e086f3cd6b67153de8671a58465fe73b2507a35bc1b2a9340be98a1523"}
Jul 23 23:01:48.145442 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:48.145402    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt" event={"ID":"b8914fd5-6a57-4ca3-88b9-b7d30837eff2","Type":"ContainerStarted","Data":"2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73"}
Jul 23 23:01:48.145902 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:48.145504    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:01:48.147098 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:48.147066    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l" event={"ID":"a4ae922a-1907-47f0-b8e9-89f77e76856e","Type":"ContainerStarted","Data":"59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610"}
Jul 23 23:01:48.147210 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:48.147120    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:48.148876 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:48.148852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5" event={"ID":"e0751b7d-9666-4640-8352-a4d72c618ca1","Type":"ContainerStarted","Data":"fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c"}
Jul 23 23:01:48.148999 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:48.148944    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:01:48.164181 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:48.164124    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt" podStartSLOduration=2.164110922 podStartE2EDuration="2.164110922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:01:48.161697034 +0000 UTC m=+2362.479225844" watchObservedRunningTime="2026-07-23 23:01:48.164110922 +0000 UTC m=+2362.481639733"
Jul 23 23:01:48.177405 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:48.177364    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l" podStartSLOduration=2.856083367 podStartE2EDuration="3.177347125s" totalImagesPullingTime="321.263758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:01:48.176916783 +0000 UTC m=+2362.494445596" watchObservedRunningTime="2026-07-23 23:01:48.177347125 +0000 UTC m=+2362.494875965"
Jul 23 23:01:48.192894 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:48.192851    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5" podStartSLOduration=2.192839823 podStartE2EDuration="2.192839823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:01:48.191367786 +0000 UTC m=+2362.508896597" watchObservedRunningTime="2026-07-23 23:01:48.192839823 +0000 UTC m=+2362.510368631"
Jul 23 23:01:51.093104 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:51.093072    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:01:51.093507 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:51.093143    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:01:54.161003 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:54.160971    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:01:59.156772 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:59.156739    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:01:59.157263 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:01:59.156803    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:02:26.324249 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:02:26.324217    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 23:02:26.328951 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:02:26.328908    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 23:06:26.533591 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:06:26.533555    2583 scope.go:117] "RemoveContainer" containerID="0aed4b8cb1fa2b43c317fa1b61875bfffc0e106d2bbb49dde2c04b215a9ee567"
Jul 23 23:06:26.542150 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:06:26.542131    2583 scope.go:117] "RemoveContainer" containerID="46369b3bb61f94e561b35dc62c71ecc2cb7e44381e0d94289e9cc9b1bc9c7485"
Jul 23 23:07:26.352588 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:26.352509    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 23:07:26.365170 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:26.365145    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 23:07:26.566564 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:26.566525    2583 scope.go:117] "RemoveContainer" containerID="78d66a76c6fb95547621a3ad174ce41d8071ec3e965d0ab8949024ff0b21c57b"
Jul 23 23:07:41.698390 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.698353    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"]
Jul 23 23:07:41.702164 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.702142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:41.704632 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.704605    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4b3ef9-serving-cert\""
Jul 23 23:07:41.712420 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.712396    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"]
Jul 23 23:07:41.719264 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.719238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76db9de8-4251-4037-b054-902be7e6303b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:41.719417 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.719272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvdxs\" (UniqueName: \"kubernetes.io/projected/76db9de8-4251-4037-b054-902be7e6303b-kube-api-access-fvdxs\") pod \"maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:41.719417 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.719301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76db9de8-4251-4037-b054-902be7e6303b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:41.820387 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.820335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76db9de8-4251-4037-b054-902be7e6303b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:41.820579 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.820508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76db9de8-4251-4037-b054-902be7e6303b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:41.820579 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.820547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvdxs\" (UniqueName: \"kubernetes.io/projected/76db9de8-4251-4037-b054-902be7e6303b-kube-api-access-fvdxs\") pod \"maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:41.821982 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.821856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76db9de8-4251-4037-b054-902be7e6303b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:41.824281 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.824254    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76db9de8-4251-4037-b054-902be7e6303b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:41.843313 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:41.843287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvdxs\" (UniqueName: \"kubernetes.io/projected/76db9de8-4251-4037-b054-902be7e6303b-kube-api-access-fvdxs\") pod \"maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:42.014971 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:42.014850    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:42.160729 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:42.160698    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"]
Jul 23 23:07:42.162500 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:07:42.162467    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76db9de8_4251_4037_b054_902be7e6303b.slice/crio-e8ebf378213dbf51c748acbee9448be994cb461bfbbce3666b9449e4a1267c73 WatchSource:0}: Error finding container e8ebf378213dbf51c748acbee9448be994cb461bfbbce3666b9449e4a1267c73: Status 404 returned error can't find the container with id e8ebf378213dbf51c748acbee9448be994cb461bfbbce3666b9449e4a1267c73
Jul 23 23:07:42.164089 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:42.164069    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 23:07:42.864715 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:42.864611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r" event={"ID":"76db9de8-4251-4037-b054-902be7e6303b","Type":"ContainerStarted","Data":"507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097"}
Jul 23 23:07:42.864715 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:42.864669    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r" event={"ID":"76db9de8-4251-4037-b054-902be7e6303b","Type":"ContainerStarted","Data":"e8ebf378213dbf51c748acbee9448be994cb461bfbbce3666b9449e4a1267c73"}
Jul 23 23:07:42.865258 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:42.864746    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:07:42.881306 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:42.881180    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r" podStartSLOduration=1.484166846 podStartE2EDuration="1.881156482s" totalImagesPullingTime="396.989636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:07:42.880834601 +0000 UTC m=+2717.198363411" watchObservedRunningTime="2026-07-23 23:07:42.881156482 +0000 UTC m=+2717.198685292"
Jul 23 23:07:48.874769 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:07:48.874731    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:08:03.806868 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:03.806827    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"]
Jul 23 23:08:03.811807 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:03.811778    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:03.814504 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:03.814479    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4f67fa-serving-cert\""
Jul 23 23:08:03.819704 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:03.819656    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"]
Jul 23 23:08:03.912352 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:03.912313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:03.912521 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:03.912363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bs6r\" (UniqueName: \"kubernetes.io/projected/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-kube-api-access-7bs6r\") pod \"maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:03.912521 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:03.912464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:04.012998 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.012957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:04.013160 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.013010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bs6r\" (UniqueName: \"kubernetes.io/projected/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-kube-api-access-7bs6r\") pod \"maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:04.013160 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.013043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:04.013673 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.013651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:04.015462 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.015438    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:04.021248 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.021225    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bs6r\" (UniqueName: \"kubernetes.io/projected/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-kube-api-access-7bs6r\") pod \"maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:04.125863 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.125827    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:04.269985 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.269894    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"]
Jul 23 23:08:04.272588 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:08:04.272542    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7651dfb7_3e4d_4f0a_a867_4d4d8bb4078f.slice/crio-b231072d821bd70c89aada4e05a31e359002ea128c4127aee9b3a303e53efe8d WatchSource:0}: Error finding container b231072d821bd70c89aada4e05a31e359002ea128c4127aee9b3a303e53efe8d: Status 404 returned error can't find the container with id b231072d821bd70c89aada4e05a31e359002ea128c4127aee9b3a303e53efe8d
Jul 23 23:08:04.413159 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.413076    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"]
Jul 23 23:08:04.417780 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.417759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:08:04.420838 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.420819    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4f67fa-dockercfg-85bx4\""
Jul 23 23:08:04.421015 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.420819    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4f67fa\""
Jul 23 23:08:04.423588 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.423299    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"]
Jul 23 23:08:04.427080 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.427060    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"]
Jul 23 23:08:04.427178 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.427163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:08:04.442367 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.442342    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"]
Jul 23 23:08:04.517368 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.517330    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5k8n\" (UniqueName: \"kubernetes.io/projected/c883d914-6a31-46c8-a115-d09a26548391-kube-api-access-f5k8n\") pod \"payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf\" (UID: \"c883d914-6a31-46c8-a115-d09a26548391\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:08:04.517368 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.517369    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjqk\" (UniqueName: \"kubernetes.io/projected/48de1262-2f57-4b2b-8bea-92946fb224ec-kube-api-access-hcjqk\") pod \"payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9\" (UID: \"48de1262-2f57-4b2b-8bea-92946fb224ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:08:04.517616 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.517437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c883d914-6a31-46c8-a115-d09a26548391-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf\" (UID: \"c883d914-6a31-46c8-a115-d09a26548391\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:08:04.517616 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.517459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48de1262-2f57-4b2b-8bea-92946fb224ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9\" (UID: \"48de1262-2f57-4b2b-8bea-92946fb224ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:08:04.618707 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.618664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5k8n\" (UniqueName: \"kubernetes.io/projected/c883d914-6a31-46c8-a115-d09a26548391-kube-api-access-f5k8n\") pod \"payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf\" (UID: \"c883d914-6a31-46c8-a115-d09a26548391\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:08:04.618707 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.618707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjqk\" (UniqueName: \"kubernetes.io/projected/48de1262-2f57-4b2b-8bea-92946fb224ec-kube-api-access-hcjqk\") pod \"payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9\" (UID: \"48de1262-2f57-4b2b-8bea-92946fb224ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:08:04.618968 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.618739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c883d914-6a31-46c8-a115-d09a26548391-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf\" (UID: \"c883d914-6a31-46c8-a115-d09a26548391\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:08:04.619014 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.618993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48de1262-2f57-4b2b-8bea-92946fb224ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9\" (UID: \"48de1262-2f57-4b2b-8bea-92946fb224ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:08:04.619522 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.619501    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c883d914-6a31-46c8-a115-d09a26548391-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf\" (UID: \"c883d914-6a31-46c8-a115-d09a26548391\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:08:04.619628 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.619609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48de1262-2f57-4b2b-8bea-92946fb224ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9\" (UID: \"48de1262-2f57-4b2b-8bea-92946fb224ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:08:04.626646 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.626615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5k8n\" (UniqueName: \"kubernetes.io/projected/c883d914-6a31-46c8-a115-d09a26548391-kube-api-access-f5k8n\") pod \"payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf\" (UID: \"c883d914-6a31-46c8-a115-d09a26548391\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:08:04.626776 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.626757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjqk\" (UniqueName: \"kubernetes.io/projected/48de1262-2f57-4b2b-8bea-92946fb224ec-kube-api-access-hcjqk\") pod \"payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9\" (UID: \"48de1262-2f57-4b2b-8bea-92946fb224ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:08:04.730500 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.730416    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:08:04.739565 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.739537    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:08:04.969161 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:04.969127    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc" event={"ID":"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f","Type":"ContainerStarted","Data":"b231072d821bd70c89aada4e05a31e359002ea128c4127aee9b3a303e53efe8d"}
Jul 23 23:08:05.077046 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:05.077020    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"]
Jul 23 23:08:05.078741 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:08:05.078703    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48de1262_2f57_4b2b_8bea_92946fb224ec.slice/crio-93d9cab620a61a2939c749d9dbc0798b57343e0b5f5975cda1dfd497adabdb75 WatchSource:0}: Error finding container 93d9cab620a61a2939c749d9dbc0798b57343e0b5f5975cda1dfd497adabdb75: Status 404 returned error can't find the container with id 93d9cab620a61a2939c749d9dbc0798b57343e0b5f5975cda1dfd497adabdb75
Jul 23 23:08:05.109258 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:05.109234    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"]
Jul 23 23:08:05.121049 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:08:05.121025    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc883d914_6a31_46c8_a115_d09a26548391.slice/crio-0a02de7705d752a96a1aa8c2bfee6303a53eb588ebcb7859484d2f1ed41f13ed WatchSource:0}: Error finding container 0a02de7705d752a96a1aa8c2bfee6303a53eb588ebcb7859484d2f1ed41f13ed: Status 404 returned error can't find the container with id 0a02de7705d752a96a1aa8c2bfee6303a53eb588ebcb7859484d2f1ed41f13ed
Jul 23 23:08:05.975832 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:05.975785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9" event={"ID":"48de1262-2f57-4b2b-8bea-92946fb224ec","Type":"ContainerStarted","Data":"2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16"}
Jul 23 23:08:05.976301 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:05.975854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9" event={"ID":"48de1262-2f57-4b2b-8bea-92946fb224ec","Type":"ContainerStarted","Data":"93d9cab620a61a2939c749d9dbc0798b57343e0b5f5975cda1dfd497adabdb75"}
Jul 23 23:08:05.976301 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:05.975889    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:08:05.977344 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:05.977313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf" event={"ID":"c883d914-6a31-46c8-a115-d09a26548391","Type":"ContainerStarted","Data":"001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1"}
Jul 23 23:08:05.977483 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:05.977366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf" event={"ID":"c883d914-6a31-46c8-a115-d09a26548391","Type":"ContainerStarted","Data":"0a02de7705d752a96a1aa8c2bfee6303a53eb588ebcb7859484d2f1ed41f13ed"}
Jul 23 23:08:05.977483 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:05.977409    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:08:05.978755 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:05.978732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc" event={"ID":"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f","Type":"ContainerStarted","Data":"764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02"}
Jul 23 23:08:05.978873 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:05.978829    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:05.994147 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:05.994019    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9" podStartSLOduration=1.994002423 podStartE2EDuration="1.994002423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:08:05.990908353 +0000 UTC m=+2740.308437162" watchObservedRunningTime="2026-07-23 23:08:05.994002423 +0000 UTC m=+2740.311531234"
Jul 23 23:08:06.007494 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:06.007449    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc" podStartSLOduration=1.479231758 podStartE2EDuration="3.007436851s" totalImagesPullingTime="1.528205093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:08:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:08:06.005966053 +0000 UTC m=+2740.323494863" watchObservedRunningTime="2026-07-23 23:08:06.007436851 +0000 UTC m=+2740.324965659"
Jul 23 23:08:06.021481 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:06.021438    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf" podStartSLOduration=2.021424624 podStartE2EDuration="2.021424624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:08:06.020703275 +0000 UTC m=+2740.338232085" watchObservedRunningTime="2026-07-23 23:08:06.021424624 +0000 UTC m=+2740.338953433"
Jul 23 23:08:11.989658 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:11.989623    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:08:16.985738 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:16.985708    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:08:16.986279 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:16.985787    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:08:25.556542 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:08:25.556507    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf_c883d914-6a31-46c8-a115-d09a26548391/payload-processing/0.log"
Jul 23 23:09:09.703919 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:09.703827    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf_c883d914-6a31-46c8-a115-d09a26548391/payload-processing/0.log"
Jul 23 23:09:13.199401 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.199364    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"]
Jul 23 23:09:13.203099 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.203071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:13.205838 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.205813    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-8300e5-serving-cert\""
Jul 23 23:09:13.214270 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.214241    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"]
Jul 23 23:09:13.322183 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.322141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:13.322407 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.322198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kngcd\" (UniqueName: \"kubernetes.io/projected/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-kube-api-access-kngcd\") pod \"maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:13.322407 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.322222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:13.423642 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.423605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:13.423841 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.423682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngcd\" (UniqueName: \"kubernetes.io/projected/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-kube-api-access-kngcd\") pod \"maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:13.423841 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.423718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:13.423841 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:09:13.423757    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-serving-cert: secret "maas-api-e2e-ipp-cleanup-8300e5-serving-cert" not found
Jul 23 23:09:13.423841 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:09:13.423825    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-maas-api-tls podName:05f671b7-42a5-4ae0-89dc-57f83f1f1cd5 nodeName:}" failed. No retries permitted until 2026-07-23 23:09:13.92380763 +0000 UTC m=+2808.241336417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k" (UID: "05f671b7-42a5-4ae0-89dc-57f83f1f1cd5") : secret "maas-api-e2e-ipp-cleanup-8300e5-serving-cert" not found
Jul 23 23:09:13.424514 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.424489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:13.435610 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.435579    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngcd\" (UniqueName: \"kubernetes.io/projected/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-kube-api-access-kngcd\") pod \"maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:13.783653 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.783611    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"]
Jul 23 23:09:13.787418 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.787391    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:13.789892 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.789867    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8300e5\""
Jul 23 23:09:13.789892 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.789867    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8300e5-dockercfg-p7gss\""
Jul 23 23:09:13.798631 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.798594    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"]
Jul 23 23:09:13.820820 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.820777    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"]
Jul 23 23:09:13.824170 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.824150    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:13.834956 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.834910    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"]
Jul 23 23:09:13.928119 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.928068    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5bw6\" (UniqueName: \"kubernetes.io/projected/129453d6-381a-40ba-b572-2fc3f2638baf-kube-api-access-h5bw6\") pod \"payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq\" (UID: \"129453d6-381a-40ba-b572-2fc3f2638baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:13.928298 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.928154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpz6q\" (UniqueName: \"kubernetes.io/projected/a0908ed8-4a76-421e-9634-5c2faed01010-kube-api-access-tpz6q\") pod \"payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9\" (UID: \"a0908ed8-4a76-421e-9634-5c2faed01010\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:13.928298 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.928228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:13.928298 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.928286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0908ed8-4a76-421e-9634-5c2faed01010-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9\" (UID: \"a0908ed8-4a76-421e-9634-5c2faed01010\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:13.928409 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.928364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129453d6-381a-40ba-b572-2fc3f2638baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq\" (UID: \"129453d6-381a-40ba-b572-2fc3f2638baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:13.930713 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:13.930680    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:14.029852 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.029813    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129453d6-381a-40ba-b572-2fc3f2638baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq\" (UID: \"129453d6-381a-40ba-b572-2fc3f2638baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:14.030387 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.029862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5bw6\" (UniqueName: \"kubernetes.io/projected/129453d6-381a-40ba-b572-2fc3f2638baf-kube-api-access-h5bw6\") pod \"payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq\" (UID: \"129453d6-381a-40ba-b572-2fc3f2638baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:14.030387 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.029892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpz6q\" (UniqueName: \"kubernetes.io/projected/a0908ed8-4a76-421e-9634-5c2faed01010-kube-api-access-tpz6q\") pod \"payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9\" (UID: \"a0908ed8-4a76-421e-9634-5c2faed01010\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:14.030387 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.030059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0908ed8-4a76-421e-9634-5c2faed01010-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9\" (UID: \"a0908ed8-4a76-421e-9634-5c2faed01010\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:14.030785 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.030759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0908ed8-4a76-421e-9634-5c2faed01010-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9\" (UID: \"a0908ed8-4a76-421e-9634-5c2faed01010\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:14.030862 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.030766    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129453d6-381a-40ba-b572-2fc3f2638baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq\" (UID: \"129453d6-381a-40ba-b572-2fc3f2638baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:14.037937 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.037863    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpz6q\" (UniqueName: \"kubernetes.io/projected/a0908ed8-4a76-421e-9634-5c2faed01010-kube-api-access-tpz6q\") pod \"payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9\" (UID: \"a0908ed8-4a76-421e-9634-5c2faed01010\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:14.038301 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.038277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5bw6\" (UniqueName: \"kubernetes.io/projected/129453d6-381a-40ba-b572-2fc3f2638baf-kube-api-access-h5bw6\") pod \"payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq\" (UID: \"129453d6-381a-40ba-b572-2fc3f2638baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:14.104672 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.104631    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:14.116129 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.116090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:14.136885 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.136848    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:14.282957 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.282874    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"]
Jul 23 23:09:14.310821 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.310785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq" event={"ID":"129453d6-381a-40ba-b572-2fc3f2638baf","Type":"ContainerStarted","Data":"17674d80778e8467d661542fc30a3ebf91e1db9edb04f26f16fa6a312472293f"}
Jul 23 23:09:14.516522 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.516495    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"]
Jul 23 23:09:14.518219 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:09:14.518194    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0908ed8_4a76_421e_9634_5c2faed01010.slice/crio-67567f77f158f50f4cf74ebe9dead61284d68044122d5b6299b3c82cd4d7d019 WatchSource:0}: Error finding container 67567f77f158f50f4cf74ebe9dead61284d68044122d5b6299b3c82cd4d7d019: Status 404 returned error can't find the container with id 67567f77f158f50f4cf74ebe9dead61284d68044122d5b6299b3c82cd4d7d019
Jul 23 23:09:14.519593 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:14.519564    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"]
Jul 23 23:09:14.520816 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:09:14.520792    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f671b7_42a5_4ae0_89dc_57f83f1f1cd5.slice/crio-b2ad85e52c4bc08b1ad97c540589cdda14b63e5a753d6e42c069fdb82a67e662 WatchSource:0}: Error finding container b2ad85e52c4bc08b1ad97c540589cdda14b63e5a753d6e42c069fdb82a67e662: Status 404 returned error can't find the container with id b2ad85e52c4bc08b1ad97c540589cdda14b63e5a753d6e42c069fdb82a67e662
Jul 23 23:09:15.320327 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:15.320286    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq" event={"ID":"129453d6-381a-40ba-b572-2fc3f2638baf","Type":"ContainerStarted","Data":"74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097"}
Jul 23 23:09:15.320777 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:15.320493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:15.322074 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:15.322051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9" event={"ID":"a0908ed8-4a76-421e-9634-5c2faed01010","Type":"ContainerStarted","Data":"82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c"}
Jul 23 23:09:15.322198 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:15.322090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9" event={"ID":"a0908ed8-4a76-421e-9634-5c2faed01010","Type":"ContainerStarted","Data":"67567f77f158f50f4cf74ebe9dead61284d68044122d5b6299b3c82cd4d7d019"}
Jul 23 23:09:15.322198 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:15.322170    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:15.323637 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:15.323612    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k" event={"ID":"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5","Type":"ContainerStarted","Data":"08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7"}
Jul 23 23:09:15.323756 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:15.323654    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k" event={"ID":"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5","Type":"ContainerStarted","Data":"b2ad85e52c4bc08b1ad97c540589cdda14b63e5a753d6e42c069fdb82a67e662"}
Jul 23 23:09:15.323756 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:15.323684    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:15.338229 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:15.338184    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq" podStartSLOduration=2.33817032 podStartE2EDuration="2.33817032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:09:15.335938167 +0000 UTC m=+2809.653466970" watchObservedRunningTime="2026-07-23 23:09:15.33817032 +0000 UTC m=+2809.655699128"
Jul 23 23:09:15.352531 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:15.352481    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k" podStartSLOduration=1.993903601 podStartE2EDuration="2.352464473s" totalImagesPullingTime="358.560872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:09:15.350844814 +0000 UTC m=+2809.668373627" watchObservedRunningTime="2026-07-23 23:09:15.352464473 +0000 UTC m=+2809.669993287"
Jul 23 23:09:15.365481 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:15.365427    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9" podStartSLOduration=2.365383161 podStartE2EDuration="2.365383161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:09:15.365271613 +0000 UTC m=+2809.682800423" watchObservedRunningTime="2026-07-23 23:09:15.365383161 +0000 UTC m=+2809.682911971"
Jul 23 23:09:21.335971 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:21.335939    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:26.332109 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:26.332078    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:26.332563 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:26.332449    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:38.970077 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:38.970039    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"]
Jul 23 23:09:38.970505 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:38.970279    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k" podUID="05f671b7-42a5-4ae0-89dc-57f83f1f1cd5" containerName="maas-api" containerID="cri-o://08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7" gracePeriod=30
Jul 23 23:09:39.017788 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.017733    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"]
Jul 23 23:09:39.018102 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.018061    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9" podUID="a0908ed8-4a76-421e-9634-5c2faed01010" containerName="payload-processing" containerID="cri-o://82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c" gracePeriod=30
Jul 23 23:09:39.032762 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.032694    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"]
Jul 23 23:09:39.033194 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.032946    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq" podUID="129453d6-381a-40ba-b572-2fc3f2638baf" containerName="payload-pre-processing" containerID="cri-o://74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097" gracePeriod=30
Jul 23 23:09:39.337406 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.337380    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:39.343665 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.343638    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:39.352841 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.352815    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:39.462482 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.462441    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kngcd\" (UniqueName: \"kubernetes.io/projected/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-kube-api-access-kngcd\") pod \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") "
Jul 23 23:09:39.462694 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.462502    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5bw6\" (UniqueName: \"kubernetes.io/projected/129453d6-381a-40ba-b572-2fc3f2638baf-kube-api-access-h5bw6\") pod \"129453d6-381a-40ba-b572-2fc3f2638baf\" (UID: \"129453d6-381a-40ba-b572-2fc3f2638baf\") "
Jul 23 23:09:39.462694 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.462615    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-openshift-service-ca\") pod \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") "
Jul 23 23:09:39.462694 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.462685    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-maas-api-tls\") pod \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\" (UID: \"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5\") "
Jul 23 23:09:39.462900 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.462730    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129453d6-381a-40ba-b572-2fc3f2638baf-plugins-config-volume\") pod \"129453d6-381a-40ba-b572-2fc3f2638baf\" (UID: \"129453d6-381a-40ba-b572-2fc3f2638baf\") "
Jul 23 23:09:39.462900 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.462807    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpz6q\" (UniqueName: \"kubernetes.io/projected/a0908ed8-4a76-421e-9634-5c2faed01010-kube-api-access-tpz6q\") pod \"a0908ed8-4a76-421e-9634-5c2faed01010\" (UID: \"a0908ed8-4a76-421e-9634-5c2faed01010\") "
Jul 23 23:09:39.462900 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.462839    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0908ed8-4a76-421e-9634-5c2faed01010-plugins-config-volume\") pod \"a0908ed8-4a76-421e-9634-5c2faed01010\" (UID: \"a0908ed8-4a76-421e-9634-5c2faed01010\") "
Jul 23 23:09:39.463105 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.463066    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05f671b7-42a5-4ae0-89dc-57f83f1f1cd5" (UID: "05f671b7-42a5-4ae0-89dc-57f83f1f1cd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:09:39.463773 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.463744    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:09:39.463884 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.463779    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0908ed8-4a76-421e-9634-5c2faed01010-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0908ed8-4a76-421e-9634-5c2faed01010" (UID: "a0908ed8-4a76-421e-9634-5c2faed01010"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:09:39.463884 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.463833    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129453d6-381a-40ba-b572-2fc3f2638baf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "129453d6-381a-40ba-b572-2fc3f2638baf" (UID: "129453d6-381a-40ba-b572-2fc3f2638baf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:09:39.465083 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.465054    2583 generic.go:356] "Generic (PLEG): container finished" podID="129453d6-381a-40ba-b572-2fc3f2638baf" containerID="74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097" exitCode=0
Jul 23 23:09:39.465196 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.465112    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"
Jul 23 23:09:39.465196 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.465165    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq" event={"ID":"129453d6-381a-40ba-b572-2fc3f2638baf","Type":"ContainerDied","Data":"74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097"}
Jul 23 23:09:39.465305 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.465240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq" event={"ID":"129453d6-381a-40ba-b572-2fc3f2638baf","Type":"ContainerDied","Data":"17674d80778e8467d661542fc30a3ebf91e1db9edb04f26f16fa6a312472293f"}
Jul 23 23:09:39.465305 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.465258    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129453d6-381a-40ba-b572-2fc3f2638baf-kube-api-access-h5bw6" (OuterVolumeSpecName: "kube-api-access-h5bw6") pod "129453d6-381a-40ba-b572-2fc3f2638baf" (UID: "129453d6-381a-40ba-b572-2fc3f2638baf"). InnerVolumeSpecName "kube-api-access-h5bw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:09:39.465305 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.465270    2583 scope.go:117] "RemoveContainer" containerID="74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097"
Jul 23 23:09:39.465534 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.465508    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05f671b7-42a5-4ae0-89dc-57f83f1f1cd5" (UID: "05f671b7-42a5-4ae0-89dc-57f83f1f1cd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 23:09:39.465624 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.465597    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-kube-api-access-kngcd" (OuterVolumeSpecName: "kube-api-access-kngcd") pod "05f671b7-42a5-4ae0-89dc-57f83f1f1cd5" (UID: "05f671b7-42a5-4ae0-89dc-57f83f1f1cd5"). InnerVolumeSpecName "kube-api-access-kngcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:09:39.465893 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.465873    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0908ed8-4a76-421e-9634-5c2faed01010-kube-api-access-tpz6q" (OuterVolumeSpecName: "kube-api-access-tpz6q") pod "a0908ed8-4a76-421e-9634-5c2faed01010" (UID: "a0908ed8-4a76-421e-9634-5c2faed01010"). InnerVolumeSpecName "kube-api-access-tpz6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:09:39.466660 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.466628    2583 generic.go:356] "Generic (PLEG): container finished" podID="a0908ed8-4a76-421e-9634-5c2faed01010" containerID="82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c" exitCode=0
Jul 23 23:09:39.466767 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.466682    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"
Jul 23 23:09:39.466767 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.466683    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9" event={"ID":"a0908ed8-4a76-421e-9634-5c2faed01010","Type":"ContainerDied","Data":"82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c"}
Jul 23 23:09:39.466767 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.466763    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9" event={"ID":"a0908ed8-4a76-421e-9634-5c2faed01010","Type":"ContainerDied","Data":"67567f77f158f50f4cf74ebe9dead61284d68044122d5b6299b3c82cd4d7d019"}
Jul 23 23:09:39.468305 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.468286    2583 generic.go:356] "Generic (PLEG): container finished" podID="05f671b7-42a5-4ae0-89dc-57f83f1f1cd5" containerID="08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7" exitCode=0
Jul 23 23:09:39.468411 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.468331    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"
Jul 23 23:09:39.468411 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.468375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k" event={"ID":"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5","Type":"ContainerDied","Data":"08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7"}
Jul 23 23:09:39.468498 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.468438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k" event={"ID":"05f671b7-42a5-4ae0-89dc-57f83f1f1cd5","Type":"ContainerDied","Data":"b2ad85e52c4bc08b1ad97c540589cdda14b63e5a753d6e42c069fdb82a67e662"}
Jul 23 23:09:39.476184 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.476141    2583 scope.go:117] "RemoveContainer" containerID="74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097"
Jul 23 23:09:39.476454 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:09:39.476423    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097\": container with ID starting with 74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097 not found: ID does not exist" containerID="74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097"
Jul 23 23:09:39.476573 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.476460    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097"} err="failed to get container status \"74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097\": rpc error: code = NotFound desc = could not find container \"74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097\": container with ID starting with 74ae4434c201eb52832875ebe58e13566f7a144cd4c25c23b4c7bb3848e5a097 not found: ID does not exist"
Jul 23 23:09:39.476573 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.476476    2583 scope.go:117] "RemoveContainer" containerID="82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c"
Jul 23 23:09:39.484635 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.484621    2583 scope.go:117] "RemoveContainer" containerID="82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c"
Jul 23 23:09:39.484886 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:09:39.484866    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c\": container with ID starting with 82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c not found: ID does not exist" containerID="82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c"
Jul 23 23:09:39.485010 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.484892    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c"} err="failed to get container status \"82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c\": rpc error: code = NotFound desc = could not find container \"82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c\": container with ID starting with 82ab05eb69c9232be293cfad81181d0263149774607973e3b8329390f1b8f04c not found: ID does not exist"
Jul 23 23:09:39.485010 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.484908    2583 scope.go:117] "RemoveContainer" containerID="08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7"
Jul 23 23:09:39.493228 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.493186    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"]
Jul 23 23:09:39.494093 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.494076    2583 scope.go:117] "RemoveContainer" containerID="08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7"
Jul 23 23:09:39.494363 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:09:39.494347    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7\": container with ID starting with 08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7 not found: ID does not exist" containerID="08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7"
Jul 23 23:09:39.494419 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.494368    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7"} err="failed to get container status \"08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7\": rpc error: code = NotFound desc = could not find container \"08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7\": container with ID starting with 08effcce7dc371b9e5bba1bedad76f43be81068a8d05dc589d702dda2508e3d7 not found: ID does not exist"
Jul 23 23:09:39.495972 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.495906    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8300e5-778d744484-l9c4k"]
Jul 23 23:09:39.505195 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.505170    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"]
Jul 23 23:09:39.509337 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.509313    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8300e5-7b7575b446-dckz9"]
Jul 23 23:09:39.564943 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.564897    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kngcd\" (UniqueName: \"kubernetes.io/projected/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-kube-api-access-kngcd\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:09:39.564943 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.564945    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5bw6\" (UniqueName: \"kubernetes.io/projected/129453d6-381a-40ba-b572-2fc3f2638baf-kube-api-access-h5bw6\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:09:39.565119 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.564958    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:09:39.565119 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.564969    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129453d6-381a-40ba-b572-2fc3f2638baf-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:09:39.565119 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.564978    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpz6q\" (UniqueName: \"kubernetes.io/projected/a0908ed8-4a76-421e-9634-5c2faed01010-kube-api-access-tpz6q\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:09:39.565119 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.564986    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0908ed8-4a76-421e-9634-5c2faed01010-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:09:39.786623 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.786593    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"]
Jul 23 23:09:39.791463 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:39.791438    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8300e5-59456bf65b-ms9vq"]
Jul 23 23:09:40.229626 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:40.229592    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f671b7-42a5-4ae0-89dc-57f83f1f1cd5" path="/var/lib/kubelet/pods/05f671b7-42a5-4ae0-89dc-57f83f1f1cd5/volumes"
Jul 23 23:09:40.230015 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:40.229965    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="129453d6-381a-40ba-b572-2fc3f2638baf" path="/var/lib/kubelet/pods/129453d6-381a-40ba-b572-2fc3f2638baf/volumes"
Jul 23 23:09:40.230292 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:09:40.230279    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0908ed8-4a76-421e-9634-5c2faed01010" path="/var/lib/kubelet/pods/a0908ed8-4a76-421e-9634-5c2faed01010/volumes"
Jul 23 23:10:09.311664 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.311614    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"]
Jul 23 23:10:09.314249 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.311862    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r" podUID="76db9de8-4251-4037-b054-902be7e6303b" containerName="maas-api" containerID="cri-o://507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097" gracePeriod=30
Jul 23 23:10:09.559508 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.559479    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:10:09.624031 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.623913    2583 generic.go:356] "Generic (PLEG): container finished" podID="76db9de8-4251-4037-b054-902be7e6303b" containerID="507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097" exitCode=0
Jul 23 23:10:09.624031 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.623986    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"
Jul 23 23:10:09.624031 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.624018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r" event={"ID":"76db9de8-4251-4037-b054-902be7e6303b","Type":"ContainerDied","Data":"507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097"}
Jul 23 23:10:09.624299 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.624086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r" event={"ID":"76db9de8-4251-4037-b054-902be7e6303b","Type":"ContainerDied","Data":"e8ebf378213dbf51c748acbee9448be994cb461bfbbce3666b9449e4a1267c73"}
Jul 23 23:10:09.624299 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.624123    2583 scope.go:117] "RemoveContainer" containerID="507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097"
Jul 23 23:10:09.633640 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.633621    2583 scope.go:117] "RemoveContainer" containerID="507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097"
Jul 23 23:10:09.633936 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:10:09.633897    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097\": container with ID starting with 507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097 not found: ID does not exist" containerID="507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097"
Jul 23 23:10:09.634050 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.633951    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097"} err="failed to get container status \"507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097\": rpc error: code = NotFound desc = could not find container \"507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097\": container with ID starting with 507edf3897098be253a28bae53201993416b80731bd2c567192f23ada4d44097 not found: ID does not exist"
Jul 23 23:10:09.733238 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.733205    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76db9de8-4251-4037-b054-902be7e6303b-openshift-service-ca\") pod \"76db9de8-4251-4037-b054-902be7e6303b\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") "
Jul 23 23:10:09.733238 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.733238    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvdxs\" (UniqueName: \"kubernetes.io/projected/76db9de8-4251-4037-b054-902be7e6303b-kube-api-access-fvdxs\") pod \"76db9de8-4251-4037-b054-902be7e6303b\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") "
Jul 23 23:10:09.733492 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.733301    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76db9de8-4251-4037-b054-902be7e6303b-maas-api-tls\") pod \"76db9de8-4251-4037-b054-902be7e6303b\" (UID: \"76db9de8-4251-4037-b054-902be7e6303b\") "
Jul 23 23:10:09.733652 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.733613    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76db9de8-4251-4037-b054-902be7e6303b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76db9de8-4251-4037-b054-902be7e6303b" (UID: "76db9de8-4251-4037-b054-902be7e6303b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:10:09.735433 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.735408    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76db9de8-4251-4037-b054-902be7e6303b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76db9de8-4251-4037-b054-902be7e6303b" (UID: "76db9de8-4251-4037-b054-902be7e6303b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 23:10:09.735522 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.735510    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76db9de8-4251-4037-b054-902be7e6303b-kube-api-access-fvdxs" (OuterVolumeSpecName: "kube-api-access-fvdxs") pod "76db9de8-4251-4037-b054-902be7e6303b" (UID: "76db9de8-4251-4037-b054-902be7e6303b"). InnerVolumeSpecName "kube-api-access-fvdxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:10:09.834163 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.834126    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76db9de8-4251-4037-b054-902be7e6303b-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:10:09.834163 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.834156    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvdxs\" (UniqueName: \"kubernetes.io/projected/76db9de8-4251-4037-b054-902be7e6303b-kube-api-access-fvdxs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:10:09.834163 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.834167    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76db9de8-4251-4037-b054-902be7e6303b-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:10:09.945807 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.945777    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"]
Jul 23 23:10:09.949184 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:09.949156    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4b3ef9-574b8bd897-fp76r"]
Jul 23 23:10:10.229581 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:10:10.229502    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76db9de8-4251-4037-b054-902be7e6303b" path="/var/lib/kubelet/pods/76db9de8-4251-4037-b054-902be7e6303b/volumes"
Jul 23 23:11:19.364555 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.364517    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"]
Jul 23 23:11:19.365252 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.364717    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc" podUID="7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f" containerName="maas-api" containerID="cri-o://764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02" gracePeriod=30
Jul 23 23:11:19.418908 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.418872    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"]
Jul 23 23:11:19.419168 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.419138    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf" podUID="c883d914-6a31-46c8-a115-d09a26548391" containerName="payload-processing" containerID="cri-o://001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1" gracePeriod=30
Jul 23 23:11:19.424533 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.424501    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"]
Jul 23 23:11:19.424771 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.424734    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9" podUID="48de1262-2f57-4b2b-8bea-92946fb224ec" containerName="payload-pre-processing" containerID="cri-o://2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16" gracePeriod=30
Jul 23 23:11:19.693634 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.693602    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:11:19.731358 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.731327    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48de1262-2f57-4b2b-8bea-92946fb224ec-plugins-config-volume\") pod \"48de1262-2f57-4b2b-8bea-92946fb224ec\" (UID: \"48de1262-2f57-4b2b-8bea-92946fb224ec\") "
Jul 23 23:11:19.731543 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.731376    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcjqk\" (UniqueName: \"kubernetes.io/projected/48de1262-2f57-4b2b-8bea-92946fb224ec-kube-api-access-hcjqk\") pod \"48de1262-2f57-4b2b-8bea-92946fb224ec\" (UID: \"48de1262-2f57-4b2b-8bea-92946fb224ec\") "
Jul 23 23:11:19.731681 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.731656    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48de1262-2f57-4b2b-8bea-92946fb224ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "48de1262-2f57-4b2b-8bea-92946fb224ec" (UID: "48de1262-2f57-4b2b-8bea-92946fb224ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:11:19.734671 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.734646    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48de1262-2f57-4b2b-8bea-92946fb224ec-kube-api-access-hcjqk" (OuterVolumeSpecName: "kube-api-access-hcjqk") pod "48de1262-2f57-4b2b-8bea-92946fb224ec" (UID: "48de1262-2f57-4b2b-8bea-92946fb224ec"). InnerVolumeSpecName "kube-api-access-hcjqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:11:19.743734 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.743709    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:11:19.762389 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.762362    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:11:19.832752 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.832717    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bs6r\" (UniqueName: \"kubernetes.io/projected/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-kube-api-access-7bs6r\") pod \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") "
Jul 23 23:11:19.832752 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.832757    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5k8n\" (UniqueName: \"kubernetes.io/projected/c883d914-6a31-46c8-a115-d09a26548391-kube-api-access-f5k8n\") pod \"c883d914-6a31-46c8-a115-d09a26548391\" (UID: \"c883d914-6a31-46c8-a115-d09a26548391\") "
Jul 23 23:11:19.833013 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.832785    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-openshift-service-ca\") pod \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") "
Jul 23 23:11:19.833013 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.832834    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-maas-api-tls\") pod \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\" (UID: \"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f\") "
Jul 23 23:11:19.833013 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.832851    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c883d914-6a31-46c8-a115-d09a26548391-plugins-config-volume\") pod \"c883d914-6a31-46c8-a115-d09a26548391\" (UID: \"c883d914-6a31-46c8-a115-d09a26548391\") "
Jul 23 23:11:19.833177 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.833151    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48de1262-2f57-4b2b-8bea-92946fb224ec-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:19.833237 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.833184    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcjqk\" (UniqueName: \"kubernetes.io/projected/48de1262-2f57-4b2b-8bea-92946fb224ec-kube-api-access-hcjqk\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:19.833300 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.833267    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f" (UID: "7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:11:19.833424 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.833398    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c883d914-6a31-46c8-a115-d09a26548391-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c883d914-6a31-46c8-a115-d09a26548391" (UID: "c883d914-6a31-46c8-a115-d09a26548391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:11:19.835104 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.835077    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-kube-api-access-7bs6r" (OuterVolumeSpecName: "kube-api-access-7bs6r") pod "7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f" (UID: "7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f"). InnerVolumeSpecName "kube-api-access-7bs6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:11:19.835104 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.835092    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c883d914-6a31-46c8-a115-d09a26548391-kube-api-access-f5k8n" (OuterVolumeSpecName: "kube-api-access-f5k8n") pod "c883d914-6a31-46c8-a115-d09a26548391" (UID: "c883d914-6a31-46c8-a115-d09a26548391"). InnerVolumeSpecName "kube-api-access-f5k8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:11:19.835259 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.835215    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f" (UID: "7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 23:11:19.934502 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.934413    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bs6r\" (UniqueName: \"kubernetes.io/projected/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-kube-api-access-7bs6r\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:19.934502 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.934443    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5k8n\" (UniqueName: \"kubernetes.io/projected/c883d914-6a31-46c8-a115-d09a26548391-kube-api-access-f5k8n\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:19.934502 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.934452    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:19.934502 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.934464    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:19.934502 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.934473    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c883d914-6a31-46c8-a115-d09a26548391-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:19.971819 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.971787    2583 generic.go:356] "Generic (PLEG): container finished" podID="48de1262-2f57-4b2b-8bea-92946fb224ec" containerID="2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16" exitCode=0
Jul 23 23:11:19.972038 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.971833    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"
Jul 23 23:11:19.972038 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.971860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9" event={"ID":"48de1262-2f57-4b2b-8bea-92946fb224ec","Type":"ContainerDied","Data":"2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16"}
Jul 23 23:11:19.972038 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.971914    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9" event={"ID":"48de1262-2f57-4b2b-8bea-92946fb224ec","Type":"ContainerDied","Data":"93d9cab620a61a2939c749d9dbc0798b57343e0b5f5975cda1dfd497adabdb75"}
Jul 23 23:11:19.972038 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.971975    2583 scope.go:117] "RemoveContainer" containerID="2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16"
Jul 23 23:11:19.973399 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.973377    2583 generic.go:356] "Generic (PLEG): container finished" podID="c883d914-6a31-46c8-a115-d09a26548391" containerID="001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1" exitCode=0
Jul 23 23:11:19.973525 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.973415    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"
Jul 23 23:11:19.973525 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.973476    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf" event={"ID":"c883d914-6a31-46c8-a115-d09a26548391","Type":"ContainerDied","Data":"001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1"}
Jul 23 23:11:19.973525 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.973509    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf" event={"ID":"c883d914-6a31-46c8-a115-d09a26548391","Type":"ContainerDied","Data":"0a02de7705d752a96a1aa8c2bfee6303a53eb588ebcb7859484d2f1ed41f13ed"}
Jul 23 23:11:19.974992 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.974967    2583 generic.go:356] "Generic (PLEG): container finished" podID="7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f" containerID="764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02" exitCode=0
Jul 23 23:11:19.975116 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.975010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc" event={"ID":"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f","Type":"ContainerDied","Data":"764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02"}
Jul 23 23:11:19.975116 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.975029    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"
Jul 23 23:11:19.975116 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.975046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc" event={"ID":"7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f","Type":"ContainerDied","Data":"b231072d821bd70c89aada4e05a31e359002ea128c4127aee9b3a303e53efe8d"}
Jul 23 23:11:19.983802 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.983781    2583 scope.go:117] "RemoveContainer" containerID="2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16"
Jul 23 23:11:19.984195 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:11:19.984134    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16\": container with ID starting with 2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16 not found: ID does not exist" containerID="2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16"
Jul 23 23:11:19.984195 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.984176    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16"} err="failed to get container status \"2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16\": rpc error: code = NotFound desc = could not find container \"2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16\": container with ID starting with 2fced61ae6d93cab5b935abce6e1d34b15cf0b699cef17936b4796a5620d4c16 not found: ID does not exist"
Jul 23 23:11:19.984337 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.984201    2583 scope.go:117] "RemoveContainer" containerID="001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1"
Jul 23 23:11:19.992479 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.992437    2583 scope.go:117] "RemoveContainer" containerID="001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1"
Jul 23 23:11:19.993123 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:11:19.992785    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1\": container with ID starting with 001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1 not found: ID does not exist" containerID="001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1"
Jul 23 23:11:19.993123 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.992848    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1"} err="failed to get container status \"001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1\": rpc error: code = NotFound desc = could not find container \"001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1\": container with ID starting with 001d0465751bc9f424c5d387618b46d166b2f864f3fec2a90208914cf0c667f1 not found: ID does not exist"
Jul 23 23:11:19.993123 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.992877    2583 scope.go:117] "RemoveContainer" containerID="764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02"
Jul 23 23:11:20.002719 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:19.997209    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"]
Jul 23 23:11:20.006288 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:20.006260    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4f67fa-6d6cbdf548-s5vr9"]
Jul 23 23:11:20.008327 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:20.008310    2583 scope.go:117] "RemoveContainer" containerID="764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02"
Jul 23 23:11:20.008639 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:11:20.008617    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02\": container with ID starting with 764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02 not found: ID does not exist" containerID="764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02"
Jul 23 23:11:20.008685 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:20.008647    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02"} err="failed to get container status \"764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02\": rpc error: code = NotFound desc = could not find container \"764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02\": container with ID starting with 764044d42877f5eaea5f7fb64c9ca523f1b75538ca1235c5ec73a9d923a66d02 not found: ID does not exist"
Jul 23 23:11:20.016150 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:20.016122    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"]
Jul 23 23:11:20.017836 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:20.017816    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4f67fa-7bf994777c-sxlrc"]
Jul 23 23:11:20.027582 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:20.027554    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"]
Jul 23 23:11:20.031155 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:20.031128    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4f67fa-59d58d89b8-lkrmf"]
Jul 23 23:11:20.229345 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:20.229261    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48de1262-2f57-4b2b-8bea-92946fb224ec" path="/var/lib/kubelet/pods/48de1262-2f57-4b2b-8bea-92946fb224ec/volumes"
Jul 23 23:11:20.229667 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:20.229654    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f" path="/var/lib/kubelet/pods/7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f/volumes"
Jul 23 23:11:20.230048 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:20.230033    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c883d914-6a31-46c8-a115-d09a26548391" path="/var/lib/kubelet/pods/c883d914-6a31-46c8-a115-d09a26548391/volumes"
Jul 23 23:11:41.379681 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.379647    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"]
Jul 23 23:11:41.380083 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.379865    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9" podUID="cc09e64a-9854-462b-829e-a0a1e10fad2d" containerName="maas-api" containerID="cri-o://a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5" gracePeriod=30
Jul 23 23:11:41.415966 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.415913    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"]
Jul 23 23:11:41.416225 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.416195    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9" podUID="90d93580-cc40-402e-9c16-ba19591d8cec" containerName="payload-processing" containerID="cri-o://2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a" gracePeriod=30
Jul 23 23:11:41.422666 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.422637    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"]
Jul 23 23:11:41.423077 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.423043    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d" podUID="6545665c-ab1a-4392-b5bd-59b5430499c9" containerName="payload-pre-processing" containerID="cri-o://e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b" gracePeriod=30
Jul 23 23:11:41.683616 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.683583    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:11:41.720972 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.720939    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:11:41.724186 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.724161    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:11:41.834687 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.834648    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc09e64a-9854-462b-829e-a0a1e10fad2d-maas-api-tls\") pod \"cc09e64a-9854-462b-829e-a0a1e10fad2d\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") "
Jul 23 23:11:41.834869 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.834702    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d93580-cc40-402e-9c16-ba19591d8cec-plugins-config-volume\") pod \"90d93580-cc40-402e-9c16-ba19591d8cec\" (UID: \"90d93580-cc40-402e-9c16-ba19591d8cec\") "
Jul 23 23:11:41.834869 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.834794    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6545665c-ab1a-4392-b5bd-59b5430499c9-plugins-config-volume\") pod \"6545665c-ab1a-4392-b5bd-59b5430499c9\" (UID: \"6545665c-ab1a-4392-b5bd-59b5430499c9\") "
Jul 23 23:11:41.834869 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.834822    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc09e64a-9854-462b-829e-a0a1e10fad2d-openshift-service-ca\") pod \"cc09e64a-9854-462b-829e-a0a1e10fad2d\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") "
Jul 23 23:11:41.835100 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.834875    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdt7n\" (UniqueName: \"kubernetes.io/projected/cc09e64a-9854-462b-829e-a0a1e10fad2d-kube-api-access-bdt7n\") pod \"cc09e64a-9854-462b-829e-a0a1e10fad2d\" (UID: \"cc09e64a-9854-462b-829e-a0a1e10fad2d\") "
Jul 23 23:11:41.835100 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.834907    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxxbd\" (UniqueName: \"kubernetes.io/projected/6545665c-ab1a-4392-b5bd-59b5430499c9-kube-api-access-fxxbd\") pod \"6545665c-ab1a-4392-b5bd-59b5430499c9\" (UID: \"6545665c-ab1a-4392-b5bd-59b5430499c9\") "
Jul 23 23:11:41.835100 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.834967    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmmvs\" (UniqueName: \"kubernetes.io/projected/90d93580-cc40-402e-9c16-ba19591d8cec-kube-api-access-lmmvs\") pod \"90d93580-cc40-402e-9c16-ba19591d8cec\" (UID: \"90d93580-cc40-402e-9c16-ba19591d8cec\") "
Jul 23 23:11:41.835250 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.835172    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d93580-cc40-402e-9c16-ba19591d8cec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90d93580-cc40-402e-9c16-ba19591d8cec" (UID: "90d93580-cc40-402e-9c16-ba19591d8cec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:11:41.835250 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.835234    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc09e64a-9854-462b-829e-a0a1e10fad2d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc09e64a-9854-462b-829e-a0a1e10fad2d" (UID: "cc09e64a-9854-462b-829e-a0a1e10fad2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:11:41.835359 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.835284    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d93580-cc40-402e-9c16-ba19591d8cec-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:41.835359 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.835324    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6545665c-ab1a-4392-b5bd-59b5430499c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6545665c-ab1a-4392-b5bd-59b5430499c9" (UID: "6545665c-ab1a-4392-b5bd-59b5430499c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:11:41.837055 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.837020    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6545665c-ab1a-4392-b5bd-59b5430499c9-kube-api-access-fxxbd" (OuterVolumeSpecName: "kube-api-access-fxxbd") pod "6545665c-ab1a-4392-b5bd-59b5430499c9" (UID: "6545665c-ab1a-4392-b5bd-59b5430499c9"). InnerVolumeSpecName "kube-api-access-fxxbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:11:41.837055 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.837029    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc09e64a-9854-462b-829e-a0a1e10fad2d-kube-api-access-bdt7n" (OuterVolumeSpecName: "kube-api-access-bdt7n") pod "cc09e64a-9854-462b-829e-a0a1e10fad2d" (UID: "cc09e64a-9854-462b-829e-a0a1e10fad2d"). InnerVolumeSpecName "kube-api-access-bdt7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:11:41.837055 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.837036    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc09e64a-9854-462b-829e-a0a1e10fad2d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc09e64a-9854-462b-829e-a0a1e10fad2d" (UID: "cc09e64a-9854-462b-829e-a0a1e10fad2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 23:11:41.837279 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.837122    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d93580-cc40-402e-9c16-ba19591d8cec-kube-api-access-lmmvs" (OuterVolumeSpecName: "kube-api-access-lmmvs") pod "90d93580-cc40-402e-9c16-ba19591d8cec" (UID: "90d93580-cc40-402e-9c16-ba19591d8cec"). InnerVolumeSpecName "kube-api-access-lmmvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:11:41.936147 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.936063    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6545665c-ab1a-4392-b5bd-59b5430499c9-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:41.936147 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.936093    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc09e64a-9854-462b-829e-a0a1e10fad2d-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:41.936147 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.936102    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdt7n\" (UniqueName: \"kubernetes.io/projected/cc09e64a-9854-462b-829e-a0a1e10fad2d-kube-api-access-bdt7n\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:41.936147 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.936111    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxxbd\" (UniqueName: \"kubernetes.io/projected/6545665c-ab1a-4392-b5bd-59b5430499c9-kube-api-access-fxxbd\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:41.936147 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.936120    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmmvs\" (UniqueName: \"kubernetes.io/projected/90d93580-cc40-402e-9c16-ba19591d8cec-kube-api-access-lmmvs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:41.936147 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:41.936131    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc09e64a-9854-462b-829e-a0a1e10fad2d-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:42.091250 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.091216    2583 generic.go:356] "Generic (PLEG): container finished" podID="cc09e64a-9854-462b-829e-a0a1e10fad2d" containerID="a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5" exitCode=0
Jul 23 23:11:42.091448 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.091264    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"
Jul 23 23:11:42.091448 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.091303    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9" event={"ID":"cc09e64a-9854-462b-829e-a0a1e10fad2d","Type":"ContainerDied","Data":"a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5"}
Jul 23 23:11:42.091448 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.091377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9" event={"ID":"cc09e64a-9854-462b-829e-a0a1e10fad2d","Type":"ContainerDied","Data":"5cdc35f4c57e0accd6fc1a8d3ca29838e79f82f8a4275a50fa6a3d24684c41f4"}
Jul 23 23:11:42.091448 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.091408    2583 scope.go:117] "RemoveContainer" containerID="a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5"
Jul 23 23:11:42.092717 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.092698    2583 generic.go:356] "Generic (PLEG): container finished" podID="90d93580-cc40-402e-9c16-ba19591d8cec" containerID="2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a" exitCode=0
Jul 23 23:11:42.092803 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.092780    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9" event={"ID":"90d93580-cc40-402e-9c16-ba19591d8cec","Type":"ContainerDied","Data":"2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a"}
Jul 23 23:11:42.092862 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.092805    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"
Jul 23 23:11:42.092862 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.092833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9" event={"ID":"90d93580-cc40-402e-9c16-ba19591d8cec","Type":"ContainerDied","Data":"41035b40abb1517ee26c4c32a433c982f9b1183d5a50b5f6a279f31fcbf05e4a"}
Jul 23 23:11:42.094148 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.094123    2583 generic.go:356] "Generic (PLEG): container finished" podID="6545665c-ab1a-4392-b5bd-59b5430499c9" containerID="e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b" exitCode=0
Jul 23 23:11:42.094266 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.094157    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"
Jul 23 23:11:42.094266 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.094211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d" event={"ID":"6545665c-ab1a-4392-b5bd-59b5430499c9","Type":"ContainerDied","Data":"e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b"}
Jul 23 23:11:42.094354 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.094261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d" event={"ID":"6545665c-ab1a-4392-b5bd-59b5430499c9","Type":"ContainerDied","Data":"a0b04cc7ea5533fc5024984dac3ba00a7aea6ff147781f3a67f8be461a10c983"}
Jul 23 23:11:42.100290 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.100267    2583 scope.go:117] "RemoveContainer" containerID="a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5"
Jul 23 23:11:42.100566 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:11:42.100548    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5\": container with ID starting with a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5 not found: ID does not exist" containerID="a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5"
Jul 23 23:11:42.100635 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.100574    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5"} err="failed to get container status \"a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5\": rpc error: code = NotFound desc = could not find container \"a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5\": container with ID starting with a851eb6c16d18ef576abb4da33d48248910d07f5722ac285733f1c8bfc37cba5 not found: ID does not exist"
Jul 23 23:11:42.100635 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.100593    2583 scope.go:117] "RemoveContainer" containerID="2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a"
Jul 23 23:11:42.110314 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.110296    2583 scope.go:117] "RemoveContainer" containerID="2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a"
Jul 23 23:11:42.110566 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:11:42.110546    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a\": container with ID starting with 2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a not found: ID does not exist" containerID="2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a"
Jul 23 23:11:42.110619 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.110573    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a"} err="failed to get container status \"2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a\": rpc error: code = NotFound desc = could not find container \"2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a\": container with ID starting with 2db3a798f685f175b9fccf8675a5d422394a4e7e67b2dac36443a9340b5faa6a not found: ID does not exist"
Jul 23 23:11:42.110619 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.110589    2583 scope.go:117] "RemoveContainer" containerID="e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b"
Jul 23 23:11:42.118494 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.118477    2583 scope.go:117] "RemoveContainer" containerID="e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b"
Jul 23 23:11:42.118745 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:11:42.118725    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b\": container with ID starting with e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b not found: ID does not exist" containerID="e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b"
Jul 23 23:11:42.118800 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.118752    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b"} err="failed to get container status \"e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b\": rpc error: code = NotFound desc = could not find container \"e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b\": container with ID starting with e24e7a357122a81a8f1946604b5a8b269ddaf76b6abf2d599265be8c9a91671b not found: ID does not exist"
Jul 23 23:11:42.119141 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.119123    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"]
Jul 23 23:11:42.124567 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.124545    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd1be5-5df5996fb-pvw7d"]
Jul 23 23:11:42.134648 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.134623    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"]
Jul 23 23:11:42.137996 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.137976    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd1be5-c56fc49d7-r5vl9"]
Jul 23 23:11:42.147275 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.147217    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"]
Jul 23 23:11:42.149332 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.149312    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd1be5-56559fccfd-qstf9"]
Jul 23 23:11:42.228956 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.228833    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6545665c-ab1a-4392-b5bd-59b5430499c9" path="/var/lib/kubelet/pods/6545665c-ab1a-4392-b5bd-59b5430499c9/volumes"
Jul 23 23:11:42.229325 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.229305    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d93580-cc40-402e-9c16-ba19591d8cec" path="/var/lib/kubelet/pods/90d93580-cc40-402e-9c16-ba19591d8cec/volumes"
Jul 23 23:11:42.229707 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:42.229689    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc09e64a-9854-462b-829e-a0a1e10fad2d" path="/var/lib/kubelet/pods/cc09e64a-9854-462b-829e-a0a1e10fad2d/volumes"
Jul 23 23:11:52.971521 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971433    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"]
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971804    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76db9de8-4251-4037-b054-902be7e6303b" containerName="maas-api"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971819    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="76db9de8-4251-4037-b054-902be7e6303b" containerName="maas-api"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971865    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0908ed8-4a76-421e-9634-5c2faed01010" containerName="payload-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971871    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0908ed8-4a76-421e-9634-5c2faed01010" containerName="payload-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971895    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f" containerName="maas-api"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971902    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f" containerName="maas-api"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971910    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d93580-cc40-402e-9c16-ba19591d8cec" containerName="payload-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971915    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d93580-cc40-402e-9c16-ba19591d8cec" containerName="payload-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971934    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6545665c-ab1a-4392-b5bd-59b5430499c9" containerName="payload-pre-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971945    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6545665c-ab1a-4392-b5bd-59b5430499c9" containerName="payload-pre-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971951    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c883d914-6a31-46c8-a115-d09a26548391" containerName="payload-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971957    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c883d914-6a31-46c8-a115-d09a26548391" containerName="payload-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971972    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48de1262-2f57-4b2b-8bea-92946fb224ec" containerName="payload-pre-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971978    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48de1262-2f57-4b2b-8bea-92946fb224ec" containerName="payload-pre-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971985    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="129453d6-381a-40ba-b572-2fc3f2638baf" containerName="payload-pre-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971990    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="129453d6-381a-40ba-b572-2fc3f2638baf" containerName="payload-pre-processing"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.971998    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f671b7-42a5-4ae0-89dc-57f83f1f1cd5" containerName="maas-api"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972003    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f671b7-42a5-4ae0-89dc-57f83f1f1cd5" containerName="maas-api"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972014    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc09e64a-9854-462b-829e-a0a1e10fad2d" containerName="maas-api"
Jul 23 23:11:52.972025 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972021    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc09e64a-9854-462b-829e-a0a1e10fad2d" containerName="maas-api"
Jul 23 23:11:52.972721 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972097    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6545665c-ab1a-4392-b5bd-59b5430499c9" containerName="payload-pre-processing"
Jul 23 23:11:52.972721 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972105    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f671b7-42a5-4ae0-89dc-57f83f1f1cd5" containerName="maas-api"
Jul 23 23:11:52.972721 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972114    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0908ed8-4a76-421e-9634-5c2faed01010" containerName="payload-processing"
Jul 23 23:11:52.972721 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972123    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="48de1262-2f57-4b2b-8bea-92946fb224ec" containerName="payload-pre-processing"
Jul 23 23:11:52.972721 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972131    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7651dfb7-3e4d-4f0a-a867-4d4d8bb4078f" containerName="maas-api"
Jul 23 23:11:52.972721 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972137    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc09e64a-9854-462b-829e-a0a1e10fad2d" containerName="maas-api"
Jul 23 23:11:52.972721 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972143    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="76db9de8-4251-4037-b054-902be7e6303b" containerName="maas-api"
Jul 23 23:11:52.972721 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972149    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="129453d6-381a-40ba-b572-2fc3f2638baf" containerName="payload-pre-processing"
Jul 23 23:11:52.972721 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972155    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d93580-cc40-402e-9c16-ba19591d8cec" containerName="payload-processing"
Jul 23 23:11:52.972721 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.972160    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c883d914-6a31-46c8-a115-d09a26548391" containerName="payload-processing"
Jul 23 23:11:52.976984 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.976966    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"
Jul 23 23:11:52.979738 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.979714    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6cq4\""
Jul 23 23:11:52.982353 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:52.982325    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"]
Jul 23 23:11:53.137128 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:53.137091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e13fb4e-4eff-4c3f-ad75-80ab6b970576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-159535-28fgd\" (UID: \"6e13fb4e-4eff-4c3f-ad75-80ab6b970576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"
Jul 23 23:11:53.238045 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:53.237947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e13fb4e-4eff-4c3f-ad75-80ab6b970576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-159535-28fgd\" (UID: \"6e13fb4e-4eff-4c3f-ad75-80ab6b970576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"
Jul 23 23:11:53.238665 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:53.238644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e13fb4e-4eff-4c3f-ad75-80ab6b970576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-159535-28fgd\" (UID: \"6e13fb4e-4eff-4c3f-ad75-80ab6b970576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"
Jul 23 23:11:53.290072 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:53.290035    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"
Jul 23 23:11:53.425262 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:53.425230    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"]
Jul 23 23:11:53.428622 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:11:53.428593    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e13fb4e_4eff_4c3f_ad75_80ab6b970576.slice/crio-a63661fbcb9c9967436a5d22264b99a2307b2b121d20fb0337b6c15851f420cc WatchSource:0}: Error finding container a63661fbcb9c9967436a5d22264b99a2307b2b121d20fb0337b6c15851f420cc: Status 404 returned error can't find the container with id a63661fbcb9c9967436a5d22264b99a2307b2b121d20fb0337b6c15851f420cc
Jul 23 23:11:54.152769 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:54.152739    2583 generic.go:356] "Generic (PLEG): container finished" podID="6e13fb4e-4eff-4c3f-ad75-80ab6b970576" containerID="4c3892397f9c2cdc5bbb8230ec3ecfac51107b9a734d2b197ab891bf4728271d" exitCode=0
Jul 23 23:11:54.153128 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:54.152824    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd" event={"ID":"6e13fb4e-4eff-4c3f-ad75-80ab6b970576","Type":"ContainerDied","Data":"4c3892397f9c2cdc5bbb8230ec3ecfac51107b9a734d2b197ab891bf4728271d"}
Jul 23 23:11:54.153128 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:54.152889    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd" event={"ID":"6e13fb4e-4eff-4c3f-ad75-80ab6b970576","Type":"ContainerStarted","Data":"a63661fbcb9c9967436a5d22264b99a2307b2b121d20fb0337b6c15851f420cc"}
Jul 23 23:11:55.294871 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:55.294848    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"
Jul 23 23:11:55.457609 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:55.457521    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e13fb4e-4eff-4c3f-ad75-80ab6b970576-maas-api-service-ca\") pod \"6e13fb4e-4eff-4c3f-ad75-80ab6b970576\" (UID: \"6e13fb4e-4eff-4c3f-ad75-80ab6b970576\") "
Jul 23 23:11:55.457913 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:55.457886    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e13fb4e-4eff-4c3f-ad75-80ab6b970576-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e13fb4e-4eff-4c3f-ad75-80ab6b970576" (UID: "6e13fb4e-4eff-4c3f-ad75-80ab6b970576"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:11:55.558909 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:55.558874    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e13fb4e-4eff-4c3f-ad75-80ab6b970576-maas-api-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:11:56.163979 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:56.163917    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd" event={"ID":"6e13fb4e-4eff-4c3f-ad75-80ab6b970576","Type":"ContainerDied","Data":"a63661fbcb9c9967436a5d22264b99a2307b2b121d20fb0337b6c15851f420cc"}
Jul 23 23:11:56.164153 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:56.164025    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a63661fbcb9c9967436a5d22264b99a2307b2b121d20fb0337b6c15851f420cc"
Jul 23 23:11:56.164153 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:11:56.163958    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"
Jul 23 23:12:03.549971 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.549918    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"]
Jul 23 23:12:03.550450 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.550160    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l" podUID="a4ae922a-1907-47f0-b8e9-89f77e76856e" containerName="maas-api" containerID="cri-o://59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610" gracePeriod=30
Jul 23 23:12:03.598130 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.598094    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"]
Jul 23 23:12:03.598416 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.598363    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt" podUID="b8914fd5-6a57-4ca3-88b9-b7d30837eff2" containerName="payload-processing" containerID="cri-o://2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73" gracePeriod=30
Jul 23 23:12:03.603503 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.603472    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"]
Jul 23 23:12:03.603765 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.603709    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5" podUID="e0751b7d-9666-4640-8352-a4d72c618ca1" containerName="payload-pre-processing" containerID="cri-o://fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c" gracePeriod=30
Jul 23 23:12:03.873192 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.873165    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:12:03.901446 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.901422    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:12:03.907028 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.907009    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:12:03.919586 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.919555    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j585c\" (UniqueName: \"kubernetes.io/projected/a4ae922a-1907-47f0-b8e9-89f77e76856e-kube-api-access-j585c\") pod \"a4ae922a-1907-47f0-b8e9-89f77e76856e\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") "
Jul 23 23:12:03.920014 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.919594    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ae922a-1907-47f0-b8e9-89f77e76856e-openshift-service-ca\") pod \"a4ae922a-1907-47f0-b8e9-89f77e76856e\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") "
Jul 23 23:12:03.920014 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.919622    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0751b7d-9666-4640-8352-a4d72c618ca1-plugins-config-volume\") pod \"e0751b7d-9666-4640-8352-a4d72c618ca1\" (UID: \"e0751b7d-9666-4640-8352-a4d72c618ca1\") "
Jul 23 23:12:03.920014 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.919682    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhrvb\" (UniqueName: \"kubernetes.io/projected/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-kube-api-access-xhrvb\") pod \"b8914fd5-6a57-4ca3-88b9-b7d30837eff2\" (UID: \"b8914fd5-6a57-4ca3-88b9-b7d30837eff2\") "
Jul 23 23:12:03.920014 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.919729    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae922a-1907-47f0-b8e9-89f77e76856e-maas-api-tls\") pod \"a4ae922a-1907-47f0-b8e9-89f77e76856e\" (UID: \"a4ae922a-1907-47f0-b8e9-89f77e76856e\") "
Jul 23 23:12:03.920014 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.919791    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-plugins-config-volume\") pod \"b8914fd5-6a57-4ca3-88b9-b7d30837eff2\" (UID: \"b8914fd5-6a57-4ca3-88b9-b7d30837eff2\") "
Jul 23 23:12:03.920014 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.919837    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmkjf\" (UniqueName: \"kubernetes.io/projected/e0751b7d-9666-4640-8352-a4d72c618ca1-kube-api-access-gmkjf\") pod \"e0751b7d-9666-4640-8352-a4d72c618ca1\" (UID: \"e0751b7d-9666-4640-8352-a4d72c618ca1\") "
Jul 23 23:12:03.921060 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.920076    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ae922a-1907-47f0-b8e9-89f77e76856e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4ae922a-1907-47f0-b8e9-89f77e76856e" (UID: "a4ae922a-1907-47f0-b8e9-89f77e76856e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:12:03.921060 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.920507    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8914fd5-6a57-4ca3-88b9-b7d30837eff2" (UID: "b8914fd5-6a57-4ca3-88b9-b7d30837eff2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:12:03.921060 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.920624    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0751b7d-9666-4640-8352-a4d72c618ca1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0751b7d-9666-4640-8352-a4d72c618ca1" (UID: "e0751b7d-9666-4640-8352-a4d72c618ca1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 23:12:03.921233 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.921063    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ae922a-1907-47f0-b8e9-89f77e76856e-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:12:03.921233 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.921083    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0751b7d-9666-4640-8352-a4d72c618ca1-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:12:03.921233 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.921096    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:12:03.922523 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.922492    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ae922a-1907-47f0-b8e9-89f77e76856e-kube-api-access-j585c" (OuterVolumeSpecName: "kube-api-access-j585c") pod "a4ae922a-1907-47f0-b8e9-89f77e76856e" (UID: "a4ae922a-1907-47f0-b8e9-89f77e76856e"). InnerVolumeSpecName "kube-api-access-j585c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:12:03.922741 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.922714    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-kube-api-access-xhrvb" (OuterVolumeSpecName: "kube-api-access-xhrvb") pod "b8914fd5-6a57-4ca3-88b9-b7d30837eff2" (UID: "b8914fd5-6a57-4ca3-88b9-b7d30837eff2"). InnerVolumeSpecName "kube-api-access-xhrvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:12:03.922946 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.922887    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ae922a-1907-47f0-b8e9-89f77e76856e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4ae922a-1907-47f0-b8e9-89f77e76856e" (UID: "a4ae922a-1907-47f0-b8e9-89f77e76856e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 23:12:03.923049 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:03.923033    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0751b7d-9666-4640-8352-a4d72c618ca1-kube-api-access-gmkjf" (OuterVolumeSpecName: "kube-api-access-gmkjf") pod "e0751b7d-9666-4640-8352-a4d72c618ca1" (UID: "e0751b7d-9666-4640-8352-a4d72c618ca1"). InnerVolumeSpecName "kube-api-access-gmkjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 23:12:04.022337 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.022298    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmkjf\" (UniqueName: \"kubernetes.io/projected/e0751b7d-9666-4640-8352-a4d72c618ca1-kube-api-access-gmkjf\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:12:04.022337 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.022332    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j585c\" (UniqueName: \"kubernetes.io/projected/a4ae922a-1907-47f0-b8e9-89f77e76856e-kube-api-access-j585c\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:12:04.022545 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.022346    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhrvb\" (UniqueName: \"kubernetes.io/projected/b8914fd5-6a57-4ca3-88b9-b7d30837eff2-kube-api-access-xhrvb\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:12:04.022545 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.022360    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae922a-1907-47f0-b8e9-89f77e76856e-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Jul 23 23:12:04.202193 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.202105    2583 generic.go:356] "Generic (PLEG): container finished" podID="b8914fd5-6a57-4ca3-88b9-b7d30837eff2" containerID="2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73" exitCode=0
Jul 23 23:12:04.202193 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.202154    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"
Jul 23 23:12:04.202193 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.202177    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt" event={"ID":"b8914fd5-6a57-4ca3-88b9-b7d30837eff2","Type":"ContainerDied","Data":"2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73"}
Jul 23 23:12:04.202452 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.202236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt" event={"ID":"b8914fd5-6a57-4ca3-88b9-b7d30837eff2","Type":"ContainerDied","Data":"63f56a2909dc329201343106fbf71c7c498cf3b97b073643a0adc8b646c0b988"}
Jul 23 23:12:04.202452 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.202252    2583 scope.go:117] "RemoveContainer" containerID="2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73"
Jul 23 23:12:04.203573 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.203544    2583 generic.go:356] "Generic (PLEG): container finished" podID="a4ae922a-1907-47f0-b8e9-89f77e76856e" containerID="59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610" exitCode=0
Jul 23 23:12:04.203703 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.203592    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"
Jul 23 23:12:04.203703 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.203627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l" event={"ID":"a4ae922a-1907-47f0-b8e9-89f77e76856e","Type":"ContainerDied","Data":"59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610"}
Jul 23 23:12:04.203703 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.203671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l" event={"ID":"a4ae922a-1907-47f0-b8e9-89f77e76856e","Type":"ContainerDied","Data":"684e66b53f4541b1c4e96498d44f678b8fa9fadf1b0226e43a7c89ecf1d4ec1a"}
Jul 23 23:12:04.205130 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.205109    2583 generic.go:356] "Generic (PLEG): container finished" podID="e0751b7d-9666-4640-8352-a4d72c618ca1" containerID="fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c" exitCode=0
Jul 23 23:12:04.205256 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.205166    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5" event={"ID":"e0751b7d-9666-4640-8352-a4d72c618ca1","Type":"ContainerDied","Data":"fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c"}
Jul 23 23:12:04.205256 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.205174    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"
Jul 23 23:12:04.205256 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.205202    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5" event={"ID":"e0751b7d-9666-4640-8352-a4d72c618ca1","Type":"ContainerDied","Data":"c64aa4e086f3cd6b67153de8671a58465fe73b2507a35bc1b2a9340be98a1523"}
Jul 23 23:12:04.212434 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.212336    2583 scope.go:117] "RemoveContainer" containerID="2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73"
Jul 23 23:12:04.212653 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:12:04.212625    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73\": container with ID starting with 2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73 not found: ID does not exist" containerID="2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73"
Jul 23 23:12:04.212738 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.212666    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73"} err="failed to get container status \"2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73\": rpc error: code = NotFound desc = could not find container \"2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73\": container with ID starting with 2fdfd37dd025e852e94acd5d20b2157b5151a2dbf3b83ef9b3852a59a6129e73 not found: ID does not exist"
Jul 23 23:12:04.212738 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.212689    2583 scope.go:117] "RemoveContainer" containerID="59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610"
Jul 23 23:12:04.221458 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.221441    2583 scope.go:117] "RemoveContainer" containerID="59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610"
Jul 23 23:12:04.221692 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:12:04.221675    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610\": container with ID starting with 59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610 not found: ID does not exist" containerID="59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610"
Jul 23 23:12:04.221771 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.221702    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610"} err="failed to get container status \"59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610\": rpc error: code = NotFound desc = could not find container \"59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610\": container with ID starting with 59ce7a49d8d9c7e32c3d4f1f1ba165136d9893b26adf700ba29dab0d3e8cb610 not found: ID does not exist"
Jul 23 23:12:04.221771 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.221722    2583 scope.go:117] "RemoveContainer" containerID="fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c"
Jul 23 23:12:04.230151 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.230130    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"]
Jul 23 23:12:04.230243 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.230164    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-159535-5bd98bdd9d-846jt"]
Jul 23 23:12:04.230588 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.230569    2583 scope.go:117] "RemoveContainer" containerID="fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c"
Jul 23 23:12:04.230815 ip-10-0-133-131 kubenswrapper[2583]: E0723 23:12:04.230799    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c\": container with ID starting with fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c not found: ID does not exist" containerID="fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c"
Jul 23 23:12:04.230872 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.230830    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c"} err="failed to get container status \"fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c\": rpc error: code = NotFound desc = could not find container \"fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c\": container with ID starting with fb24f0351831cfe25853ae32d9a2ce4b5506a05885e2d69d0acd1d8c91f91e0c not found: ID does not exist"
Jul 23 23:12:04.239036 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.239013    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"]
Jul 23 23:12:04.240587 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.240551    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-159535-59f995b595-kc28l"]
Jul 23 23:12:04.250496 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.250475    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"]
Jul 23 23:12:04.254177 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:04.254156    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-159535-7f64c87bd-62vf5"]
Jul 23 23:12:06.228501 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:06.228468    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4ae922a-1907-47f0-b8e9-89f77e76856e" path="/var/lib/kubelet/pods/a4ae922a-1907-47f0-b8e9-89f77e76856e/volumes"
Jul 23 23:12:06.228851 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:06.228814    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8914fd5-6a57-4ca3-88b9-b7d30837eff2" path="/var/lib/kubelet/pods/b8914fd5-6a57-4ca3-88b9-b7d30837eff2/volumes"
Jul 23 23:12:06.229150 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:06.229137    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0751b7d-9666-4640-8352-a4d72c618ca1" path="/var/lib/kubelet/pods/e0751b7d-9666-4640-8352-a4d72c618ca1/volumes"
Jul 23 23:12:08.037623 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:08.037585    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"]
Jul 23 23:12:08.040321 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:08.040291    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-159535-28fgd"]
Jul 23 23:12:08.228192 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:08.228163    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e13fb4e-4eff-4c3f-ad75-80ab6b970576" path="/var/lib/kubelet/pods/6e13fb4e-4eff-4c3f-ad75-80ab6b970576/volumes"
Jul 23 23:12:21.390682 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:21.390651    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7845c898b6-sphkp_60cdae2a-9da2-4f71-91a9-729226c067fe/manager/0.log"
Jul 23 23:12:21.768968 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:21.768828    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fc7cc8c86-kzm29_49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b/manager/0.log"
Jul 23 23:12:21.782537 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:21.782514    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fc7cc8c86-kzm29_49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b/copy-manifests/0.log"
Jul 23 23:12:22.507756 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:22.507732    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747445-76whr_a0f6172e-8bf6-44cd-b94b-0153a2552a97/cleanup/0.log"
Jul 23 23:12:25.455205 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:25.455173    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wm4lz_7ead33c0-068d-4bd6-9348-cadd6918c921/discovery/0.log"
Jul 23 23:12:26.194243 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:26.194214    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84db8994fd-csdvv_f88b4c99-967b-4ded-8e2d-00bf67380e09/router/0.log"
Jul 23 23:12:26.384348 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:26.384320    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 23:12:26.399129 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:26.399105    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 23:12:26.677734 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:26.677709    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jh4db_9ee20b09-3e01-4352-9c14-442ba6bc44f3/main/0.log"
Jul 23 23:12:26.921401 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:26.921294    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc2qr7_5445b100-83bb-4857-8f99-9c12589f86b5/main/0.log"
Jul 23 23:12:35.778451 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:35.778410    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mtsxj_729340b2-7e6e-474a-8475-db80f5a1ac8b/global-pull-secret-syncer/0.log"
Jul 23 23:12:35.995053 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:35.995021    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kwlkg_68c30bcd-6fb8-4d32-adcc-881a10cdc158/konnectivity-agent/0.log"
Jul 23 23:12:36.014872 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:36.014832    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-131.ec2.internal_63161ea256ddec4bd72e56dccd7997d2/haproxy/0.log"
Jul 23 23:12:43.331389 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:43.331357    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-h2t79_031164ea-f2ff-4883-80bb-30979f504bb6/cluster-monitoring-operator/0.log"
Jul 23 23:12:43.753775 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:43.753748    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vv2b5_d81ce0b1-47e7-43ea-b8bb-da8395263376/node-exporter/0.log"
Jul 23 23:12:43.775549 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:43.775522    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vv2b5_d81ce0b1-47e7-43ea-b8bb-da8395263376/kube-rbac-proxy/0.log"
Jul 23 23:12:43.795449 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:43.795423    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vv2b5_d81ce0b1-47e7-43ea-b8bb-da8395263376/init-textfile/0.log"
Jul 23 23:12:45.548063 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:45.548032    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-tfzqx_36d92e8d-d0d3-4b4a-aa52-f67a77ccea41/networking-console-plugin/0.log"
Jul 23 23:12:46.181288 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181249    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"]
Jul 23 23:12:46.181628 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181616    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0751b7d-9666-4640-8352-a4d72c618ca1" containerName="payload-pre-processing"
Jul 23 23:12:46.181672 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181630    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0751b7d-9666-4640-8352-a4d72c618ca1" containerName="payload-pre-processing"
Jul 23 23:12:46.181672 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181644    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ae922a-1907-47f0-b8e9-89f77e76856e" containerName="maas-api"
Jul 23 23:12:46.181672 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181649    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ae922a-1907-47f0-b8e9-89f77e76856e" containerName="maas-api"
Jul 23 23:12:46.181672 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181659    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8914fd5-6a57-4ca3-88b9-b7d30837eff2" containerName="payload-processing"
Jul 23 23:12:46.181672 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181666    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8914fd5-6a57-4ca3-88b9-b7d30837eff2" containerName="payload-processing"
Jul 23 23:12:46.181828 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181676    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e13fb4e-4eff-4c3f-ad75-80ab6b970576" containerName="revoke-keys"
Jul 23 23:12:46.181828 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181682    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e13fb4e-4eff-4c3f-ad75-80ab6b970576" containerName="revoke-keys"
Jul 23 23:12:46.181828 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181741    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e13fb4e-4eff-4c3f-ad75-80ab6b970576" containerName="revoke-keys"
Jul 23 23:12:46.181828 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181748    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8914fd5-6a57-4ca3-88b9-b7d30837eff2" containerName="payload-processing"
Jul 23 23:12:46.181828 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181754    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ae922a-1907-47f0-b8e9-89f77e76856e" containerName="maas-api"
Jul 23 23:12:46.181828 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.181760    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0751b7d-9666-4640-8352-a4d72c618ca1" containerName="payload-pre-processing"
Jul 23 23:12:46.184817 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.184790    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.185414 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.185385    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/2.log"
Jul 23 23:12:46.187313 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.187290    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbzbm\"/\"openshift-service-ca.crt\""
Jul 23 23:12:46.188557 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.188527    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbzbm\"/\"kube-root-ca.crt\""
Jul 23 23:12:46.188677 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.188572    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vbzbm\"/\"default-dockercfg-5tpj8\""
Jul 23 23:12:46.190571 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.190516    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-tnv5g_b29adb35-eeca-46dd-b18e-8a3cb48d4769/console-operator/3.log"
Jul 23 23:12:46.194459 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.194437    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"]
Jul 23 23:12:46.200968 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.200944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-podres\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.201059 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.201004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-lib-modules\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.201059 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.201044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-proc\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.201179 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.201158    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-sys\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.201241 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.201221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jrzs\" (UniqueName: \"kubernetes.io/projected/21dfce05-bff4-4341-b683-3b86b02df80a-kube-api-access-2jrzs\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.302152 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.302113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-podres\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.302368 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.302175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-lib-modules\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.302368 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.302210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-proc\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.302368 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.302289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-podres\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.302368 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.302289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-sys\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.302368 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.302338    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-sys\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.302368 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.302339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-lib-modules\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.302694 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.302375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21dfce05-bff4-4341-b683-3b86b02df80a-proc\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.302694 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.302394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrzs\" (UniqueName: \"kubernetes.io/projected/21dfce05-bff4-4341-b683-3b86b02df80a-kube-api-access-2jrzs\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.310892 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.310870    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrzs\" (UniqueName: \"kubernetes.io/projected/21dfce05-bff4-4341-b683-3b86b02df80a-kube-api-access-2jrzs\") pod \"perf-node-gather-daemonset-slp4w\" (UID: \"21dfce05-bff4-4341-b683-3b86b02df80a\") " pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.496882 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.496781    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:46.627530 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:46.627505    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"]
Jul 23 23:12:46.629496 ip-10-0-133-131 kubenswrapper[2583]: W0723 23:12:46.629461    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod21dfce05_bff4_4341_b683_3b86b02df80a.slice/crio-d929f96d40d7d9aaf64cac61f4324219a32c0ea7d204b271866b5b7711e61705 WatchSource:0}: Error finding container d929f96d40d7d9aaf64cac61f4324219a32c0ea7d204b271866b5b7711e61705: Status 404 returned error can't find the container with id d929f96d40d7d9aaf64cac61f4324219a32c0ea7d204b271866b5b7711e61705
Jul 23 23:12:47.403067 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:47.403028    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w" event={"ID":"21dfce05-bff4-4341-b683-3b86b02df80a","Type":"ContainerStarted","Data":"8716bf666085cf21b66902ae00528f38cf557516bb9c2153fa32b3dc50161079"}
Jul 23 23:12:47.403255 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:47.403081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w" event={"ID":"21dfce05-bff4-4341-b683-3b86b02df80a","Type":"ContainerStarted","Data":"d929f96d40d7d9aaf64cac61f4324219a32c0ea7d204b271866b5b7711e61705"}
Jul 23 23:12:47.403255 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:47.403111    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:47.420960 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:47.420890    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w" podStartSLOduration=1.420871571 podStartE2EDuration="1.420871571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:12:47.420290481 +0000 UTC m=+3021.737819292" watchObservedRunningTime="2026-07-23 23:12:47.420871571 +0000 UTC m=+3021.738400426"
Jul 23 23:12:47.616065 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:47.616030    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-85wbm_5ae231e2-f5bf-4933-ad63-9c81b4931ce2/volume-data-source-validator/0.log"
Jul 23 23:12:48.600338 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:48.600311    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lnlzn_3a3df64e-97c8-4a17-8848-6fd2c8fd4057/dns/0.log"
Jul 23 23:12:48.621327 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:48.621301    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lnlzn_3a3df64e-97c8-4a17-8848-6fd2c8fd4057/kube-rbac-proxy/0.log"
Jul 23 23:12:48.730142 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:48.730114    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-68765_89ffe12b-74bf-4582-8395-b47766839015/dns-node-resolver/0.log"
Jul 23 23:12:49.307986 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:49.307957    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4xjxj_889cb48e-740e-4093-9908-67f62a225840/node-ca/0.log"
Jul 23 23:12:50.447641 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:50.447603    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wm4lz_7ead33c0-068d-4bd6-9348-cadd6918c921/discovery/0.log"
Jul 23 23:12:50.667159 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:50.667129    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84db8994fd-csdvv_f88b4c99-967b-4ded-8e2d-00bf67380e09/router/0.log"
Jul 23 23:12:51.493203 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:51.493166    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-78sqp_72439e60-4ee5-4235-bd03-68c54f40aa4e/serve-healthcheck-canary/0.log"
Jul 23 23:12:51.970167 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:51.970131    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-gbh2j_4b018493-e500-4c1e-9b5c-88285f6a9dc4/insights-operator/0.log"
Jul 23 23:12:51.970501 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:51.970485    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-gbh2j_4b018493-e500-4c1e-9b5c-88285f6a9dc4/insights-operator/1.log"
Jul 23 23:12:52.127138 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:52.127110    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7mbjs_c4e6016d-f94a-41a4-bbc1-4303d8158985/kube-rbac-proxy/0.log"
Jul 23 23:12:52.146940 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:52.146907    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7mbjs_c4e6016d-f94a-41a4-bbc1-4303d8158985/exporter/0.log"
Jul 23 23:12:52.168120 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:52.168072    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7mbjs_c4e6016d-f94a-41a4-bbc1-4303d8158985/extractor/0.log"
Jul 23 23:12:53.416733 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:53.416706    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vbzbm/perf-node-gather-daemonset-slp4w"
Jul 23 23:12:54.534669 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:54.534637    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7845c898b6-sphkp_60cdae2a-9da2-4f71-91a9-729226c067fe/manager/0.log"
Jul 23 23:12:54.720239 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:54.720198    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fc7cc8c86-kzm29_49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b/manager/0.log"
Jul 23 23:12:54.737646 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:54.737620    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fc7cc8c86-kzm29_49a5bb1e-bb5d-4188-84b4-bcb0f53a3c9b/copy-manifests/0.log"
Jul 23 23:12:56.231100 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:56.231066    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58df94799c-gqwq4_5916f2f0-298c-425d-9a2d-4c1e7b2fdebf/manager/0.log"
Jul 23 23:12:57.050990 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:57.050955    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wm4lz_7ead33c0-068d-4bd6-9348-cadd6918c921/discovery/0.log"
Jul 23 23:12:57.224406 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:57.224380    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84db8994fd-csdvv_f88b4c99-967b-4ded-8e2d-00bf67380e09/router/0.log"
Jul 23 23:12:58.821300 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:58.821261    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wm4lz_7ead33c0-068d-4bd6-9348-cadd6918c921/discovery/0.log"
Jul 23 23:12:59.009552 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:12:59.009517    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84db8994fd-csdvv_f88b4c99-967b-4ded-8e2d-00bf67380e09/router/0.log"
Jul 23 23:13:01.372393 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:01.372362    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-xgwvt_3e6516e0-bcc1-4cf1-ac64-2b1f0179c9f5/migrator/0.log"
Jul 23 23:13:01.395342 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:01.395314    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-xgwvt_3e6516e0-bcc1-4cf1-ac64-2b1f0179c9f5/graceful-termination/0.log"
Jul 23 23:13:01.738133 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:01.738059    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-6pdgh_1e2ff89d-860c-44a9-b40b-983d106daea3/kube-storage-version-migrator-operator/1.log"
Jul 23 23:13:01.739104 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:01.739076    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-6pdgh_1e2ff89d-860c-44a9-b40b-983d106daea3/kube-storage-version-migrator-operator/0.log"
Jul 23 23:13:02.735893 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:02.735860    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2bk9g_051cb605-0708-4f32-bf6e-1e687d328a0a/kube-multus/0.log"
Jul 23 23:13:03.102145 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:03.102118    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w5kzg_cd941d5f-38b3-49b4-a92e-afcec5348fb2/kube-multus-additional-cni-plugins/0.log"
Jul 23 23:13:03.121593 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:03.121569    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w5kzg_cd941d5f-38b3-49b4-a92e-afcec5348fb2/egress-router-binary-copy/0.log"
Jul 23 23:13:03.140698 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:03.140673    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w5kzg_cd941d5f-38b3-49b4-a92e-afcec5348fb2/cni-plugins/0.log"
Jul 23 23:13:03.159678 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:03.159651    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w5kzg_cd941d5f-38b3-49b4-a92e-afcec5348fb2/bond-cni-plugin/0.log"
Jul 23 23:13:03.180139 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:03.180116    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w5kzg_cd941d5f-38b3-49b4-a92e-afcec5348fb2/routeoverride-cni/0.log"
Jul 23 23:13:03.201379 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:03.201356    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w5kzg_cd941d5f-38b3-49b4-a92e-afcec5348fb2/whereabouts-cni-bincopy/0.log"
Jul 23 23:13:03.221348 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:03.221320    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w5kzg_cd941d5f-38b3-49b4-a92e-afcec5348fb2/whereabouts-cni/0.log"
Jul 23 23:13:03.280385 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:03.280341    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d6b9c_6d068205-8a0a-4ae6-b8a2-f831bf0e9752/network-metrics-daemon/0.log"
Jul 23 23:13:03.297321 ip-10-0-133-131 kubenswrapper[2583]: I0723 23:13:03.297293    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d6b9c_6d068205-8a0a-4ae6-b8a2-f831bf0e9752/kube-rbac-proxy/0.log"