Jul 31 21:35:59.599433 ip-10-0-135-241 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 21:36:00.052118 ip-10-0-135-241 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 21:36:00.052118 ip-10-0-135-241 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 21:36:00.052118 ip-10-0-135-241 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 21:36:00.052118 ip-10-0-135-241 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 21:36:00.052118 ip-10-0-135-241 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 21:36:00.052899 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.052204    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 21:36:00.057033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057017    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 21:36:00.057033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057032    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 21:36:00.057033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057036    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057039    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057043    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057046    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057049    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057052    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057055    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057058    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057066    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057069    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057072    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057075    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057078    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057081    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057092    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057095    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057097    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057100    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057103    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057105    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 21:36:00.057161 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057108    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057111    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057113    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057116    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057119    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057121    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057124    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057127    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057129    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057144    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057148    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057152    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057156    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057160    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057166    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057170    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057173    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057175    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057178    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057181    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 21:36:00.057898 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.057218    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 21:36:00.059804 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059790    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 21:36:00.059804 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059803    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059807    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059812    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059815    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059819    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059823    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059826    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059829    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059833    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059836    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059839    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059843    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059849    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059854    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059858    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059862    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059865    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059869    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 21:36:00.059868 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059873    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059878    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059883    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059886    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059889    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059896    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059899    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059902    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059905    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059907    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059910    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059914    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059917    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059928    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059931    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059934    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059937    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059943    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059947    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059950    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 21:36:00.060310 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059953    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059956    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059959    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059962    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059965    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059968    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059971    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059973    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059976    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059981    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059984    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059987    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059991    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059995    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.059998    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.060002    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 21:36:00.060781 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.060005    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 21:36:00.061201 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061189    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 21:36:00.061201 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061200    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061204    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061208    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061211    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061213    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061216    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061219    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061222    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061228    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061234    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061237    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061240    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061243    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061246    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061249    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061252    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061255    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061258    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061261    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 21:36:00.061256 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061264    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061267    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061272    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061275    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061277    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061280    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061283    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061286    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061289    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061292    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061295    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061297    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061301    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061304    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061310    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061314    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061318    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061321    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061324    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061327    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 21:36:00.061721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061330    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061333    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061336    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061339    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061342    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061345    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061348    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061353    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061356    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061359    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061362    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061364    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061367    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061371    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061374    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061377    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061379    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061382    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061385    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 21:36:00.062225 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061390    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061393    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061396    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061399    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061403    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061407    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061410    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061413    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061417    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061422    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061425    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061428    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061434    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061437    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061439    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061442    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061445    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061448    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061451    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 21:36:00.062688 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061453    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061456    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061459    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061461    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061464    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061469    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061472    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061475    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061478    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061481    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061484    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061487    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061489    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061492    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061495    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061498    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061500    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061503    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061508    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061511    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 21:36:00.063163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.061513    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062227    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062234    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062237    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062243    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062246    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062250    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062254    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062258    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062269    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062274    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062277    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062280    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062283    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062286    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062289    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062294    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062297    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062300    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062303    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 21:36:00.063664 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062306    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062309    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062312    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062315    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062318    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062321    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062323    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062326    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062331    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062334    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062337    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062339    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062342    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062345    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062348    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062351    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062354    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062356    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062359    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062361    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 21:36:00.064143 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062364    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062369    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062377    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062380    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062383    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062385    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062388    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062391    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062394    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062396    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062400    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062402    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062405    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062408    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062413    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062415    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062418    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062420    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062423    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062425    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 21:36:00.064618 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062428    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062431    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062433    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062436    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062439    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062441    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062448    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062452    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062455    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062458    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062461    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062463    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062466    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062469    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062471    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062479    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062482    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062485    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062490    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 21:36:00.065171 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062493    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062496    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062498    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062501    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062504    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062507    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062510    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062514    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062516    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062519    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062522    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062526    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062529    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062532    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062534    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062537    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062542    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062545    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062549    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 21:36:00.065642 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062552    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.062557    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062895    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062912    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062926    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062931    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062935    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062938    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062943    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062947    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062951    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062954    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062958    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062961    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062964    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062967    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062970    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062975    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062979    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062984    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.062993    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063000    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063003    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063008    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 21:36:00.066092 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063015    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063028    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063031    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063035    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063038    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063041    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063044    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063047    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063051    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063053    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063058    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063062    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063065    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063068    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063071    2561 flags.go:64] FLAG: --enable-server="true"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063073    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063078    2561 flags.go:64] FLAG: --event-burst="100"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063082    2561 flags.go:64] FLAG: --event-qps="50"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063085    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063088    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063091    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063094    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063097    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063100    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063103    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 31 21:36:00.066715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063106    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063109    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063112    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063115    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063117    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063120    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063123    2561 flags.go:64] FLAG: --feature-gates=""
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063126    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063130    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063144    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063148    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063151    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063154    2561 flags.go:64] FLAG: --help="false"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063157    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063161    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063163    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063167    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063170    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063175    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063178    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063180    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063183    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063186    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 21:36:00.067336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063189    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063192    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063195    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063198    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063201    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063204    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063207    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063209    2561 flags.go:64] FLAG: --lock-file=""
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063212    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063215    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063218    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063223    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063226    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063229    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063231    2561 flags.go:64] FLAG: --logging-format="text"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063234    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063237    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063240    2561 flags.go:64] FLAG: --manifest-url=""
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063244    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063249    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063252    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063265    2561 flags.go:64] FLAG: --max-pods="110"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063268    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063271    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063274    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 21:36:00.067926 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063277    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063280    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063283    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063286    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063294    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063297    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063300    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063303    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063306    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063311    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063314    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063317    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063320    2561 flags.go:64] FLAG: --port="10250"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063323    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063326    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-046960123da3a63e5"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063329    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063332    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063335    2561 flags.go:64] FLAG: --register-node="true"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063338    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063341    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063344    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063347    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063349    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063353    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063356    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 21:36:00.068608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063359    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063362    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063365    2561 flags.go:64] FLAG: --runonce="false"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063368    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063376    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063379    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063382    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063385    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063388    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063391    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063394    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063397    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063400    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063403    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063406    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063409    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063412    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063414    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063420    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063423    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063426    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063430    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063432    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063435    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063438    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 21:36:00.069243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063441    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063444    2561 flags.go:64] FLAG: --v="2"
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063448    2561 flags.go:64] FLAG: --version="false"
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063454    2561 flags.go:64] FLAG: --vmodule=""
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063458    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.063461    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063600    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063604    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063610    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063613    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063615    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063618    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063622    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063625    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063628    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063630    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063635    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063638    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063642    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063645    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063647    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 21:36:00.069835 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063650    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063653    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063655    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063658    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063662    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063665    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063668    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063672    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063675    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063678    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063681    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063683    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063686    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063689    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063691    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063694    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063697    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063699    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063702    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 21:36:00.070691 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063706    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063709    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063711    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063714    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063716    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063720    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063722    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063725    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063728    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063730    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063733    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063736    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063738    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063741    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063743    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063746    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063749    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063751    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063754    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063756    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063759    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 21:36:00.071499 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063761    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063764    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063766    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063769    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063772    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063775    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063777    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063780    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063782    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063784    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063789    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063792    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063795    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063798    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063801    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063804    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063806    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063811    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063814    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 21:36:00.072389 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063817    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063819    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063822    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063825    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063828    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063830    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063833    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063836    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063838    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063841    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063843    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063846    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063849    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063851    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063854    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063856    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063859    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063861    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063864    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063867    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 21:36:00.073308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063869    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063872    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063875    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063878    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.063882    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.064509    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.072497    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.072517    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.072536    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.072546    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072637    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072646    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072650    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072656    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072660    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 21:36:00.074160 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072666    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072673    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072679    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072684    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072689    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072693    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072698    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072702    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072707    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072711    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072718    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072724    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072728    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072733    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072737    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072741    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072746    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072750    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072755    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 21:36:00.074532 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072759    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072763    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072768    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072772    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072776    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072783    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072788    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072793    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072797    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072801    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072805    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072810    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072814    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072819    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072823    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072827    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072831    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072835    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072839    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072844    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 21:36:00.075217 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072848    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072853    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072857    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072861    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072865    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072869    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072874    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072878    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072883    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072887    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072891    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072895    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072899    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072903    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072907    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072911    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072916    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072921    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072927    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072932    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 21:36:00.076033 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072936    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072940    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072944    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072948    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072953    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072959    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072965    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072969    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072974    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072978    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072983    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072987    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072992    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.072996    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073001    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073005    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073010    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073014    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073019    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 21:36:00.076650 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073023    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073027    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073031    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073035    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073039    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073043    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073048    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073052    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073056    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073061    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073066    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073070    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073075    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073079    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073091    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 21:36:00.077305 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073096    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.073105    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073328    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073339    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073344    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073349    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073354    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073358    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073363    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073367    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073371    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073377    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073381    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073386    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073390    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 21:36:00.077680 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073393    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073398    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073402    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073407    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073411    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073416    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073421    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073426    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073430    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073434    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073439    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073443    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073447    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073451    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073456    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073471    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073476    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073481    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073485    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073489    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 21:36:00.078087 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073493    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073497    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073502    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073506    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073510    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073514    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073518    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073523    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073527    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073534    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073540    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073545    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073549    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073554    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073558    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073562    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073566    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073571    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073575    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 21:36:00.078607 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073579    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073583    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073589    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073595    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073600    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073605    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073610    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073615    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073619    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073628    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073632    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073637    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073641    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073645    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073651    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073657    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073661    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073665    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073669    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 21:36:00.079128 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073674    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073678    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073682    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073687    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073691    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073695    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073700    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073705    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073709    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073713    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073717    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073722    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073726    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073730    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073734    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073739    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073744    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073748    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073752    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073756    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 21:36:00.079598 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073760    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073764    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073769    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073781    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073786    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073791    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073795    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:00.073799    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.073807    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.073970    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.077093    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.078826    2561 server.go:1019] "Starting client certificate rotation"
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.078945    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 21:36:00.080151 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.078976    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 21:36:00.082786 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.082770    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 21:36:00.087148 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.087120    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 21:36:00.096956 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.096940    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 31 21:36:00.100981 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.100962    2561 log.go:25] "Validated CRI v1 image API"
Jul 31 21:36:00.101836 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.101818    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 21:36:00.104374 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.104354    2561 fs.go:135] Filesystem UUIDs: map[2da57c37-54f5-4ac4-a76d-e5f353a9bd98:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b34b0686-f654-4afa-b857-645244c9383e:/dev/nvme0n1p3]
Jul 31 21:36:00.104452 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.104373    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 21:36:00.110304 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.110283    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 21:36:00.111020 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.110912    2561 manager.go:216] Machine: {Timestamp:2026-07-31 21:36:00.108553803 +0000 UTC m=+0.371302159 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100946 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c9f2f3cecea986f47737f2aecb75c SystemUUID:ec2c9f2f-3cec-ea98-6f47-737f2aecb75c BootID:4a3a4f99-23cd-40be-8a88-142052603afe Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2f:38:97:e8:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2f:38:97:e8:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:f5:a7:9a:61:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 21:36:00.111020 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.111016    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 21:36:00.111147 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.111124    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 21:36:00.114792 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.114762    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 21:36:00.114929 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.114794    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 21:36:00.115635 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.115625    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 21:36:00.115669 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.115639    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 21:36:00.115669 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.115647    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 21:36:00.115669 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.115666    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 21:36:00.116848 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.116839    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 21:36:00.122172 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.122160    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 21:36:00.122295 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.122285    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 21:36:00.122327 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.122321    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 21:36:00.122364 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.122335    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 21:36:00.122364 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.122348    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 21:36:00.122364 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.122356    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 31 21:36:00.122458 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.122366    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 21:36:00.122809 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.122797    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 21:36:00.122852 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.122815    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 21:36:00.125318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.125299    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 21:36:00.127407 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.127393    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 21:36:00.127460 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.127425    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 21:36:00.128903 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128891    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 21:36:00.128942 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128907    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 21:36:00.128942 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128914    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 21:36:00.128942 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128919    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 21:36:00.128942 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128929    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 21:36:00.128942 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128937    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 21:36:00.129074 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128947    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 21:36:00.129074 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128953    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 21:36:00.129074 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128960    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 21:36:00.129074 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128965    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 21:36:00.129074 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128974    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 21:36:00.129074 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.128979    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 21:36:00.129838 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.129826    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 21:36:00.129870 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.129841    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 21:36:00.131675 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.131644    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 21:36:00.131675 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.131647    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 21:36:00.131984 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.131951    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-241.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 21:36:00.135900 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.135884    2561 server.go:1268] "Started kubelet"
Jul 31 21:36:00.135995 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.135931    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 21:36:00.136048 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.136017    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 21:36:00.136101 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.136061    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 21:36:00.136702 ip-10-0-135-241 systemd[1]: Started Kubernetes Kubelet.
Jul 31 21:36:00.138067 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.138052    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 21:36:00.140171 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.140154    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 21:36:00.145252 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.141728    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-241.ec2.internal.18c77d7ea359fd71  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-241.ec2.internal,UID:ip-10-0-135-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-241.ec2.internal,},FirstTimestamp:2026-07-31 21:36:00.135626097 +0000 UTC m=+0.398374452,LastTimestamp:2026-07-31 21:36:00.135626097 +0000 UTC m=+0.398374452,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-241.ec2.internal,}"
Jul 31 21:36:00.146719 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.146693    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 21:36:00.149110 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.149093    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 21:36:00.149184 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.149105    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 21:36:00.149292 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.149272    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:00.150187 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150168    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 21:36:00.150187 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150189    2561 factory.go:55] Registering systemd factory
Jul 31 21:36:00.150324 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150199    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 31 21:36:00.150324 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150294    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 21:36:00.150324 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150306    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 21:36:00.150460 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150379    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 21:36:00.150525 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150474    2561 factory.go:153] Registering CRI-O factory
Jul 31 21:36:00.150525 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150488    2561 factory.go:223] Registration of the crio container factory successfully
Jul 31 21:36:00.150525 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150494    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 21:36:00.150525 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150502    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 21:36:00.150525 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150519    2561 factory.go:103] Registering Raw factory
Jul 31 21:36:00.150748 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150534    2561 manager.go:1196] Started watching for new ooms in manager
Jul 31 21:36:00.150984 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.150972    2561 manager.go:349] Starting recovery of all containers
Jul 31 21:36:00.152701 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.152671    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 21:36:00.152824 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.152802    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 21:36:00.161815 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.161798    2561 manager.go:354] Recovery completed
Jul 31 21:36:00.166417 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.166402    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 21:36:00.168221 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.168202    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 21:36:00.168298 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.168234    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 21:36:00.168298 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.168245    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientPID"
Jul 31 21:36:00.168801 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.168788    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 21:36:00.168801 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.168798    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 21:36:00.168922 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.168812    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 21:36:00.168922 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.168842    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 21:36:00.170118 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.170050    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-241.ec2.internal.18c77d7ea54b5623  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-241.ec2.internal,UID:ip-10-0-135-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-241.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-241.ec2.internal,},FirstTimestamp:2026-07-31 21:36:00.168220195 +0000 UTC m=+0.430968550,LastTimestamp:2026-07-31 21:36:00.168220195 +0000 UTC m=+0.430968550,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-241.ec2.internal,}"
Jul 31 21:36:00.171089 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.171076    2561 policy_none.go:49] "None policy: Start"
Jul 31 21:36:00.171180 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.171092    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 21:36:00.171180 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.171106    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 21:36:00.171964 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.171949    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 21:36:00.172009 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.171975    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 21:36:00.172009 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.171985    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientPID"
Jul 31 21:36:00.173146 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.173121    2561 policy_none.go:47] "Start"
Jul 31 21:36:00.182454 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.182380    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-241.ec2.internal.18c77d7ea54ba0a0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-241.ec2.internal,UID:ip-10-0-135-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-241.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-241.ec2.internal,},FirstTimestamp:2026-07-31 21:36:00.168239264 +0000 UTC m=+0.430987622,LastTimestamp:2026-07-31 21:36:00.168239264 +0000 UTC m=+0.430987622,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-241.ec2.internal,}"
Jul 31 21:36:00.192770 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.192703    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-241.ec2.internal.18c77d7ea54bc58e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-241.ec2.internal,UID:ip-10-0-135-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-241.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-241.ec2.internal,},FirstTimestamp:2026-07-31 21:36:00.168248718 +0000 UTC m=+0.430997076,LastTimestamp:2026-07-31 21:36:00.168248718 +0000 UTC m=+0.430997076,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-241.ec2.internal,}"
Jul 31 21:36:00.201800 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.201731    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-241.ec2.internal.18c77d7ea54b5623\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-241.ec2.internal.18c77d7ea54b5623  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-241.ec2.internal,UID:ip-10-0-135-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-241.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-241.ec2.internal,},FirstTimestamp:2026-07-31 21:36:00.168220195 +0000 UTC m=+0.430968550,LastTimestamp:2026-07-31 21:36:00.171963294 +0000 UTC m=+0.434711648,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-241.ec2.internal,}"
Jul 31 21:36:00.209946 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.209922    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t9hp2"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.213341    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-241.ec2.internal.18c77d7ea54ba0a0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-241.ec2.internal.18c77d7ea54ba0a0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-241.ec2.internal,UID:ip-10-0-135-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-241.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-241.ec2.internal,},FirstTimestamp:2026-07-31 21:36:00.168239264 +0000 UTC m=+0.430987622,LastTimestamp:2026-07-31 21:36:00.171980098 +0000 UTC m=+0.434728452,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-241.ec2.internal,}"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.213850    2561 manager.go:337] "Starting Device Plugin manager"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.213880    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.213890    2561 server.go:87] "Starting device plugin registration server"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.214180    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.214194    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.214291    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.214378    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.214386    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.214967    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.215019    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.216436    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t9hp2"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.219918    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.220993    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.221030    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.221052    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.221095    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.223786    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Jul 31 21:36:00.231978 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.223817    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-241.ec2.internal.18c77d7ea54bc58e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-241.ec2.internal.18c77d7ea54bc58e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-241.ec2.internal,UID:ip-10-0-135-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-241.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-241.ec2.internal,},FirstTimestamp:2026-07-31 21:36:00.168248718 +0000 UTC m=+0.430997076,LastTimestamp:2026-07-31 21:36:00.171989004 +0000 UTC m=+0.434737360,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-241.ec2.internal,}"
Jul 31 21:36:00.315322 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.315089    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 21:36:00.316217 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.316201    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 21:36:00.316318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.316229    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 21:36:00.316318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.316239    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientPID"
Jul 31 21:36:00.316318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.316262    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.321959 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.321926    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-241.ec2.internal"]
Jul 31 21:36:00.322009 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.322000    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 21:36:00.322798 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.322783    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 21:36:00.322886 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.322808    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 21:36:00.322886 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.322820    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientPID"
Jul 31 21:36:00.323343 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.323331    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.323380 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.323352    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-241.ec2.internal\": node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:00.325106 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.325093    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 21:36:00.325253 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.325237    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.325314 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.325276    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 21:36:00.325825 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.325807    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 21:36:00.325888 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.325841    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 21:36:00.325888 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.325849    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 21:36:00.325888 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.325856    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientPID"
Jul 31 21:36:00.325888 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.325867    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 21:36:00.325888 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.325881    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientPID"
Jul 31 21:36:00.328774 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.328757    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.328861 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.328785    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 21:36:00.329465 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.329447    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 21:36:00.329553 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.329478    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 21:36:00.329553 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.329492    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientPID"
Jul 31 21:36:00.356684 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.356661    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:00.358822 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.358807    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-241.ec2.internal\" not found" node="ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.363156 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.363128    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-241.ec2.internal\" not found" node="ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.451620 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.451589    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/74b4ba3bfec4eba5141be9bc0bf6ece8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal\" (UID: \"74b4ba3bfec4eba5141be9bc0bf6ece8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.451620 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.451622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74b4ba3bfec4eba5141be9bc0bf6ece8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal\" (UID: \"74b4ba3bfec4eba5141be9bc0bf6ece8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.451824 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.451645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b97b9740959824207498b954ade89901-config\") pod \"kube-apiserver-proxy-ip-10-0-135-241.ec2.internal\" (UID: \"b97b9740959824207498b954ade89901\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.457675 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.457655    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:00.552225 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.552197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/74b4ba3bfec4eba5141be9bc0bf6ece8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal\" (UID: \"74b4ba3bfec4eba5141be9bc0bf6ece8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.552339 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.552228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74b4ba3bfec4eba5141be9bc0bf6ece8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal\" (UID: \"74b4ba3bfec4eba5141be9bc0bf6ece8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.552339 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.552245    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b97b9740959824207498b954ade89901-config\") pod \"kube-apiserver-proxy-ip-10-0-135-241.ec2.internal\" (UID: \"b97b9740959824207498b954ade89901\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.552339 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.552286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b97b9740959824207498b954ade89901-config\") pod \"kube-apiserver-proxy-ip-10-0-135-241.ec2.internal\" (UID: \"b97b9740959824207498b954ade89901\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.552339 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.552297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/74b4ba3bfec4eba5141be9bc0bf6ece8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal\" (UID: \"74b4ba3bfec4eba5141be9bc0bf6ece8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.552339 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.552321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74b4ba3bfec4eba5141be9bc0bf6ece8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal\" (UID: \"74b4ba3bfec4eba5141be9bc0bf6ece8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.558293 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.558270    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:00.659171 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.659078    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:00.662251 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.662232    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.666910 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:00.666886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:00.759736 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.759695    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:00.860775 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.860741    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:00.961859 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:00.961760    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:01.063016 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:01.062979    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:01.078269 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.078231    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 21:36:01.095152 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.095125    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 21:36:01.163670 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:01.163637    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:01.188967 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.188940    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 21:36:01.190163 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.190102    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 21:36:01.190285 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.190167    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 21:36:01.190285 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.190186    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeHasSufficientPID"
Jul 31 21:36:01.222972 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.222889    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 21:31:00 +0000 UTC" deadline="2028-02-15 06:21:06.040542645 +0000 UTC"
Jul 31 21:36:01.222972 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.222925    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13520h45m4.817621502s"
Jul 31 21:36:01.264735 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:01.264697    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:01.356987 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.356957    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 21:36:01.364974 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:01.364949    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-241.ec2.internal\" not found"
Jul 31 21:36:01.388796 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.388773    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 21:36:01.394172 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:01.394125    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97b9740959824207498b954ade89901.slice/crio-ac3ba12b2864ed47133ad965d3184ce7bbc0437d7f754b863a9c248378b76bfa WatchSource:0}: Error finding container ac3ba12b2864ed47133ad965d3184ce7bbc0437d7f754b863a9c248378b76bfa: Status 404 returned error can't find the container with id ac3ba12b2864ed47133ad965d3184ce7bbc0437d7f754b863a9c248378b76bfa
Jul 31 21:36:01.394792 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:01.394771    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b4ba3bfec4eba5141be9bc0bf6ece8.slice/crio-a90e0e9ad74fcee46511b50677621bfad76bdb69556af752923c824d607fa62d WatchSource:0}: Error finding container a90e0e9ad74fcee46511b50677621bfad76bdb69556af752923c824d607fa62d: Status 404 returned error can't find the container with id a90e0e9ad74fcee46511b50677621bfad76bdb69556af752923c824d607fa62d
Jul 31 21:36:01.399664 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.399647    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 21:36:01.451757 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.451716    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:01.467726 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.467692    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 21:36:01.470352 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.469987    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-241.ec2.internal"
Jul 31 21:36:01.480129 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.480072    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 21:36:01.521389 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:01.521364    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 21:36:02.124047 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.124005    2561 apiserver.go:52] "Watching apiserver"
Jul 31 21:36:02.131316 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.131287    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 21:36:02.131705 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.131681    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-6wcq8","openshift-multus/multus-mh7vc","openshift-ovn-kubernetes/ovnkube-node-qfwgj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b","openshift-dns/node-resolver-d2sf8","openshift-image-registry/node-ca-sgfk2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal","openshift-multus/multus-additional-cni-plugins-jk845","openshift-multus/network-metrics-daemon-fpcs6","openshift-network-diagnostics/network-check-target-lwcrg","openshift-network-operator/iptables-alerter-dptpt","kube-system/global-pull-secret-syncer-rw7xc","kube-system/konnectivity-agent-nfmj9","kube-system/kube-apiserver-proxy-ip-10-0-135-241.ec2.internal"]
Jul 31 21:36:02.135130 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.135108    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.137815 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.137796    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-42gq9\""
Jul 31 21:36:02.138227 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.137959    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 21:36:02.138227 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.138050    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 21:36:02.138227 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.138066    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 21:36:02.138456 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.138256    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dptpt"
Jul 31 21:36:02.138456 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.138320    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 21:36:02.138456 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.138361    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 21:36:02.140191 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.140169    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.142478 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.142458    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hsppq\""
Jul 31 21:36:02.142600 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.142578    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.142676 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.142618    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 21:36:02.142738 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.142592    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 21:36:02.143368 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.143214    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 21:36:02.143368 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.143300    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 21:36:02.143762 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.143738    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 21:36:02.144166 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.144128    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rz4fr\""
Jul 31 21:36:02.144278 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.144218    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 21:36:02.144353 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.144307    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 21:36:02.145507 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.145466    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 21:36:02.145840 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.145815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d2sf8"
Jul 31 21:36:02.146124 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.146097    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 21:36:02.146569 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.146550    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 21:36:02.146661 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.146624    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 21:36:02.146784 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.146630    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hknz5\""
Jul 31 21:36:02.148277 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.148173    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 21:36:02.148277 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.148182    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 21:36:02.148843 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.148817    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z86rc\""
Jul 31 21:36:02.149708 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.149687    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 21:36:02.149890 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.149873    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:02.149964 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.149943    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:02.152448 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.152426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.154938 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.154803    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 21:36:02.154938 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.154820    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 21:36:02.155136 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.155090    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:02.156014 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.155606    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 21:36:02.156014 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.155644    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:02.156014 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.155276    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tmhdb\""
Jul 31 21:36:02.159751 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.159671    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sgfk2"
Jul 31 21:36:02.161645 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.161627    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 21:36:02.161740 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.161650    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 21:36:02.162406 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.162387    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 21:36:02.162535 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.162521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.163705 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.163681    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2lc78\""
Jul 31 21:36:02.164478 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-kubelet\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.164574 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-host\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.164574 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164522    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-cni-bin\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.164574 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164548    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-cnibin\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.164791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164572    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-run-netns\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.164791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164596    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74q4g\" (UniqueName: \"kubernetes.io/projected/c173f8c2-063d-45d3-9a97-f83842088f2b-kube-api-access-74q4g\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.164791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-sysctl-d\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.164791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpmzs\" (UniqueName: \"kubernetes.io/projected/f13d700b-a730-4cf1-b3c7-00f71811a639-kube-api-access-rpmzs\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.164791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-run-ovn\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.164791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c173f8c2-063d-45d3-9a97-f83842088f2b-ovnkube-script-lib\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.164791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164721    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-var-lib-kubelet\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.164791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164746    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q84cd\""
Jul 31 21:36:02.164791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-tuned\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g87gk\" (UniqueName: \"kubernetes.io/projected/cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d-kube-api-access-g87gk\") pod \"node-resolver-d2sf8\" (UID: \"cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d\") " pod="openshift-dns/node-resolver-d2sf8"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-etc-selinux\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164861    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-device-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164906    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-os-release\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-run\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.164985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f13d700b-a730-4cf1-b3c7-00f71811a639-tmp\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d-tmp-dir\") pod \"node-resolver-d2sf8\" (UID: \"cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d\") " pod="openshift-dns/node-resolver-d2sf8"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165167    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c173f8c2-063d-45d3-9a97-f83842088f2b-env-overrides\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165209    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-socket-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-sys-fs\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-var-lib-openvswitch\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.165294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165306    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-modprobe-d\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-sys\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-kubernetes\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165382    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165414    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c173f8c2-063d-45d3-9a97-f83842088f2b-ovn-node-metrics-cert\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165431    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-registration-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-system-cni-dir\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb35c507-56b4-4340-b132-bb962698c791-host-slash\") pod \"iptables-alerter-dptpt\" (UID: \"cb35c507-56b4-4340-b132-bb962698c791\") " pod="openshift-network-operator/iptables-alerter-dptpt"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c173f8c2-063d-45d3-9a97-f83842088f2b-ovnkube-config\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165542    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-sysctl-conf\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d-hosts-file\") pod \"node-resolver-d2sf8\" (UID: \"cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d\") " pod="openshift-dns/node-resolver-d2sf8"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mpj\" (UniqueName: \"kubernetes.io/projected/cb35c507-56b4-4340-b132-bb962698c791-kube-api-access-d9mpj\") pod \"iptables-alerter-dptpt\" (UID: \"cb35c507-56b4-4340-b132-bb962698c791\") " pod="openshift-network-operator/iptables-alerter-dptpt"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-slash\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165647    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-etc-openvswitch\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-lib-modules\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-run-systemd\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.166095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8gb7\" (UniqueName: \"kubernetes.io/projected/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-kube-api-access-d8gb7\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnmq6\" (UniqueName: \"kubernetes.io/projected/228725d4-468e-4b0c-b209-fc141baf250b-kube-api-access-pnmq6\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165791    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-node-log\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.165795    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-cni-netd\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-run-openvswitch\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-log-socket\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-systemd\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.165994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb35c507-56b4-4340-b132-bb962698c791-iptables-alerter-script\") pod \"iptables-alerter-dptpt\" (UID: \"cb35c507-56b4-4340-b132-bb962698c791\") " pod="openshift-network-operator/iptables-alerter-dptpt"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.166025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-systemd-units\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.166060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-sysconfig\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.166098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2tvh\" (UniqueName: \"kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh\") pod \"network-check-target-lwcrg\" (UID: \"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e\") " pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:02.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.166121    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-cni-binary-copy\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.168884 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.168865    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:02.169837 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.169777    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 21:36:02.173118 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.173100    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 21:36:02.173392 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.173376    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j6dwf\""
Jul 31 21:36:02.173586 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.173560    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 21:36:02.200705 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.200672    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wr4bb"
Jul 31 21:36:02.210463 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.210437    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wr4bb"
Jul 31 21:36:02.227734 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.227686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-241.ec2.internal" event={"ID":"b97b9740959824207498b954ade89901","Type":"ContainerStarted","Data":"ac3ba12b2864ed47133ad965d3184ce7bbc0437d7f754b863a9c248378b76bfa"}
Jul 31 21:36:02.229254 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.229228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal" event={"ID":"74b4ba3bfec4eba5141be9bc0bf6ece8","Type":"ContainerStarted","Data":"a90e0e9ad74fcee46511b50677621bfad76bdb69556af752923c824d607fa62d"}
Jul 31 21:36:02.251964 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.251898    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 21:36:02.266670 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-run-ovn\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.266904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c173f8c2-063d-45d3-9a97-f83842088f2b-ovnkube-script-lib\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.266904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-var-lib-kubelet\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.266904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-tuned\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.266904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g87gk\" (UniqueName: \"kubernetes.io/projected/cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d-kube-api-access-g87gk\") pod \"node-resolver-d2sf8\" (UID: \"cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d\") " pod="openshift-dns/node-resolver-d2sf8"
Jul 31 21:36:02.266904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-etc-selinux\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.266904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2b9\" (UniqueName: \"kubernetes.io/projected/00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e-kube-api-access-qd2b9\") pod \"node-ca-sgfk2\" (UID: \"00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e\") " pod="openshift-image-registry/node-ca-sgfk2"
Jul 31 21:36:02.266904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-var-lib-cni-multus\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.266904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-device-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.266904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266900    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-os-release\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-run\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266945    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f13d700b-a730-4cf1-b3c7-00f71811a639-tmp\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.266969    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d-tmp-dir\") pod \"node-resolver-d2sf8\" (UID: \"cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d\") " pod="openshift-dns/node-resolver-d2sf8"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267012    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-system-cni-dir\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-daemon-config\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267109    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267151    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c173f8c2-063d-45d3-9a97-f83842088f2b-env-overrides\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-socket-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-sys-fs\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d760f73-be89-4905-88ed-68554584ed68-dbus\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267268    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-var-lib-openvswitch\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.267411 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267406    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-modprobe-d\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-sys\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-kubernetes\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267487    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-cni-binary-copy\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-var-lib-kubelet\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c173f8c2-063d-45d3-9a97-f83842088f2b-ovn-node-metrics-cert\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267678    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-etc-selinux\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-registration-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267792    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e-host\") pod \"node-ca-sgfk2\" (UID: \"00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e\") " pod="openshift-image-registry/node-ca-sgfk2"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3166d1ee-0352-46a8-a585-0189cba32bd6-konnectivity-ca\") pod \"konnectivity-agent-nfmj9\" (UID: \"3166d1ee-0352-46a8-a585-0189cba32bd6\") " pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-system-cni-dir\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb35c507-56b4-4340-b132-bb962698c791-host-slash\") pod \"iptables-alerter-dptpt\" (UID: \"cb35c507-56b4-4340-b132-bb962698c791\") " pod="openshift-network-operator/iptables-alerter-dptpt"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c173f8c2-063d-45d3-9a97-f83842088f2b-ovnkube-config\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267973    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-sysctl-conf\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d-hosts-file\") pod \"node-resolver-d2sf8\" (UID: \"cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d\") " pod="openshift-dns/node-resolver-d2sf8"
Jul 31 21:36:02.268299 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3166d1ee-0352-46a8-a585-0189cba32bd6-agent-certs\") pod \"konnectivity-agent-nfmj9\" (UID: \"3166d1ee-0352-46a8-a585-0189cba32bd6\") " pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mpj\" (UniqueName: \"kubernetes.io/projected/cb35c507-56b4-4340-b132-bb962698c791-kube-api-access-d9mpj\") pod \"iptables-alerter-dptpt\" (UID: \"cb35c507-56b4-4340-b132-bb962698c791\") " pod="openshift-network-operator/iptables-alerter-dptpt"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-slash\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268157    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-slash\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-sys-fs\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-etc-openvswitch\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-lib-modules\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb35c507-56b4-4340-b132-bb962698c791-host-slash\") pod \"iptables-alerter-dptpt\" (UID: \"cb35c507-56b4-4340-b132-bb962698c791\") " pod="openshift-network-operator/iptables-alerter-dptpt"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268265    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-device-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-run-systemd\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-registration-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-socket-dir-parent\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-run\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-conf-dir\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8gb7\" (UniqueName: \"kubernetes.io/projected/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-kube-api-access-d8gb7\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnmq6\" (UniqueName: \"kubernetes.io/projected/228725d4-468e-4b0c-b209-fc141baf250b-kube-api-access-pnmq6\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268515    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whwqm\" (UniqueName: \"kubernetes.io/projected/6d70e996-a47e-4761-b0c0-0795bc2bd2be-kube-api-access-whwqm\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:02.269143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-cni-dir\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-run-k8s-cni-cncf-io\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268607    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-run-netns\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268632    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-run-multus-certs\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-node-log\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268690    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-cni-netd\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-run-openvswitch\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-log-socket\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-systemd\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-var-lib-cni-bin\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d760f73-be89-4905-88ed-68554584ed68-kubelet-config\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-var-lib-kubelet\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb35c507-56b4-4340-b132-bb962698c791-iptables-alerter-script\") pod \"iptables-alerter-dptpt\" (UID: \"cb35c507-56b4-4340-b132-bb962698c791\") " pod="openshift-network-operator/iptables-alerter-dptpt"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268926    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-systemd-units\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c173f8c2-063d-45d3-9a97-f83842088f2b-ovnkube-config\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-sysconfig\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.270091 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268978    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c173f8c2-063d-45d3-9a97-f83842088f2b-ovnkube-script-lib\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tvh\" (UniqueName: \"kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh\") pod \"network-check-target-lwcrg\" (UID: \"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e\") " pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e-serviceca\") pod \"node-ca-sgfk2\" (UID: \"00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e\") " pod="openshift-image-registry/node-ca-sgfk2"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-run-ovn\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-cni-binary-copy\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-kubelet\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-host\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-sysctl-conf\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-hostroot\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-sysconfig\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.267266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-os-release\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-systemd-units\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d-hosts-file\") pod \"node-resolver-d2sf8\" (UID: \"cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d\") " pod="openshift-dns/node-resolver-d2sf8"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.268229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-system-cni-dir\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-cni-netd\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-etc-kubernetes\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.270966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269435    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-run-openvswitch\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269439    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb35c507-56b4-4340-b132-bb962698c791-iptables-alerter-script\") pod \"iptables-alerter-dptpt\" (UID: \"cb35c507-56b4-4340-b132-bb962698c791\") " pod="openshift-network-operator/iptables-alerter-dptpt"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xqj6\" (UniqueName: \"kubernetes.io/projected/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-kube-api-access-6xqj6\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-cni-bin\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-systemd\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-cnibin\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-run-netns\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269549    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269551    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-sys\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74q4g\" (UniqueName: \"kubernetes.io/projected/c173f8c2-063d-45d3-9a97-f83842088f2b-kube-api-access-74q4g\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269608    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-sysctl-d\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269628    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-var-lib-openvswitch\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmzs\" (UniqueName: \"kubernetes.io/projected/f13d700b-a730-4cf1-b3c7-00f71811a639-kube-api-access-rpmzs\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269668    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-modprobe-d\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-cnibin\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269723    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-host\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.271965 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-os-release\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-log-socket\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269912    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228725d4-468e-4b0c-b209-fc141baf250b-socket-dir\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-kubernetes\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.269455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-run-systemd\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-node-log\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-run-netns\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-cni-bin\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-cnibin\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-host-kubelet\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270709    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c173f8c2-063d-45d3-9a97-f83842088f2b-etc-openvswitch\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270808    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-lib-modules\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-cni-binary-copy\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d-tmp-dir\") pod \"node-resolver-d2sf8\" (UID: \"cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d\") " pod="openshift-dns/node-resolver-d2sf8"
Jul 31 21:36:02.274001 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.270933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-sysctl-d\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.275101 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.271110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c173f8c2-063d-45d3-9a97-f83842088f2b-env-overrides\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.275101 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.271686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f13d700b-a730-4cf1-b3c7-00f71811a639-tmp\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.275101 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.272338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f13d700b-a730-4cf1-b3c7-00f71811a639-etc-tuned\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.275101 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.273979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c173f8c2-063d-45d3-9a97-f83842088f2b-ovn-node-metrics-cert\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.278085 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.278028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g87gk\" (UniqueName: \"kubernetes.io/projected/cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d-kube-api-access-g87gk\") pod \"node-resolver-d2sf8\" (UID: \"cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d\") " pod="openshift-dns/node-resolver-d2sf8"
Jul 31 21:36:02.280893 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.280151    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 21:36:02.280893 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.280177    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 21:36:02.280893 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.280192    2561 projected.go:196] Error preparing data for projected volume kube-api-access-x2tvh for pod openshift-network-diagnostics/network-check-target-lwcrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:02.280893 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.280260    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh podName:f626a3ca-8916-4bc6-8c74-71fa3b2cee8e nodeName:}" failed. No retries permitted until 2026-07-31 21:36:02.780237138 +0000 UTC m=+3.042985502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x2tvh" (UniqueName: "kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh") pod "network-check-target-lwcrg" (UID: "f626a3ca-8916-4bc6-8c74-71fa3b2cee8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:02.280893 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.280830    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mpj\" (UniqueName: \"kubernetes.io/projected/cb35c507-56b4-4340-b132-bb962698c791-kube-api-access-d9mpj\") pod \"iptables-alerter-dptpt\" (UID: \"cb35c507-56b4-4340-b132-bb962698c791\") " pod="openshift-network-operator/iptables-alerter-dptpt"
Jul 31 21:36:02.282324 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.282297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnmq6\" (UniqueName: \"kubernetes.io/projected/228725d4-468e-4b0c-b209-fc141baf250b-kube-api-access-pnmq6\") pod \"aws-ebs-csi-driver-node-mjd8b\" (UID: \"228725d4-468e-4b0c-b209-fc141baf250b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.282837 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.282788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8gb7\" (UniqueName: \"kubernetes.io/projected/9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc-kube-api-access-d8gb7\") pod \"multus-additional-cni-plugins-jk845\" (UID: \"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc\") " pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.284487 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.284440    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74q4g\" (UniqueName: \"kubernetes.io/projected/c173f8c2-063d-45d3-9a97-f83842088f2b-kube-api-access-74q4g\") pod \"ovnkube-node-qfwgj\" (UID: \"c173f8c2-063d-45d3-9a97-f83842088f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.284635 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.284615    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmzs\" (UniqueName: \"kubernetes.io/projected/f13d700b-a730-4cf1-b3c7-00f71811a639-kube-api-access-rpmzs\") pod \"tuned-6wcq8\" (UID: \"f13d700b-a730-4cf1-b3c7-00f71811a639\") " pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.370045 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e-host\") pod \"node-ca-sgfk2\" (UID: \"00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e\") " pod="openshift-image-registry/node-ca-sgfk2"
Jul 31 21:36:02.370045 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3166d1ee-0352-46a8-a585-0189cba32bd6-konnectivity-ca\") pod \"konnectivity-agent-nfmj9\" (UID: \"3166d1ee-0352-46a8-a585-0189cba32bd6\") " pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:02.370318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3166d1ee-0352-46a8-a585-0189cba32bd6-agent-certs\") pod \"konnectivity-agent-nfmj9\" (UID: \"3166d1ee-0352-46a8-a585-0189cba32bd6\") " pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:02.370318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-socket-dir-parent\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-conf-dir\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e-host\") pod \"node-ca-sgfk2\" (UID: \"00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e\") " pod="openshift-image-registry/node-ca-sgfk2"
Jul 31 21:36:02.370318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwqm\" (UniqueName: \"kubernetes.io/projected/6d70e996-a47e-4761-b0c0-0795bc2bd2be-kube-api-access-whwqm\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:02.370318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-cni-dir\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-run-k8s-cni-cncf-io\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-run-netns\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370280    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-socket-dir-parent\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-run-multus-certs\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-var-lib-cni-bin\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-run-k8s-cni-cncf-io\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d760f73-be89-4905-88ed-68554584ed68-kubelet-config\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e-serviceca\") pod \"node-ca-sgfk2\" (UID: \"00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e\") " pod="openshift-image-registry/node-ca-sgfk2"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-hostroot\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-cni-dir\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370492    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-etc-kubernetes\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-hostroot\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-run-netns\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370601    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-etc-kubernetes\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-run-multus-certs\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xqj6\" (UniqueName: \"kubernetes.io/projected/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-kube-api-access-6xqj6\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-conf-dir\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370679    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d760f73-be89-4905-88ed-68554584ed68-kubelet-config\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-var-lib-cni-bin\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-cnibin\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3166d1ee-0352-46a8-a585-0189cba32bd6-konnectivity-ca\") pod \"konnectivity-agent-nfmj9\" (UID: \"3166d1ee-0352-46a8-a585-0189cba32bd6\") " pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-os-release\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.370960 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-cnibin\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.370985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e-serviceca\") pod \"node-ca-sgfk2\" (UID: \"00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e\") " pod="openshift-image-registry/node-ca-sgfk2"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371004    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-os-release\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2b9\" (UniqueName: \"kubernetes.io/projected/00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e-kube-api-access-qd2b9\") pod \"node-ca-sgfk2\" (UID: \"00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e\") " pod="openshift-image-registry/node-ca-sgfk2"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-var-lib-cni-multus\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-system-cni-dir\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-var-lib-cni-multus\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-daemon-config\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371208    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d760f73-be89-4905-88ed-68554584ed68-dbus\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.371226    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371237    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-cni-binary-copy\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-system-cni-dir\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.371295    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs podName:6d70e996-a47e-4761-b0c0-0795bc2bd2be nodeName:}" failed. No retries permitted until 2026-07-31 21:36:02.871275772 +0000 UTC m=+3.134024134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs") pod "network-metrics-daemon-fpcs6" (UID: "6d70e996-a47e-4761-b0c0-0795bc2bd2be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371336    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d760f73-be89-4905-88ed-68554584ed68-dbus\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-var-lib-kubelet\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-host-var-lib-kubelet\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.371849 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.371528    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:02.372500 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.371595    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret podName:2d760f73-be89-4905-88ed-68554584ed68 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:02.871576138 +0000 UTC m=+3.134324503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret") pod "global-pull-secret-syncer-rw7xc" (UID: "2d760f73-be89-4905-88ed-68554584ed68") : object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:02.372500 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-multus-daemon-config\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.372500 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.371719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-cni-binary-copy\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.373349 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.373321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3166d1ee-0352-46a8-a585-0189cba32bd6-agent-certs\") pod \"konnectivity-agent-nfmj9\" (UID: \"3166d1ee-0352-46a8-a585-0189cba32bd6\") " pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:02.380842 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.380786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2b9\" (UniqueName: \"kubernetes.io/projected/00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e-kube-api-access-qd2b9\") pod \"node-ca-sgfk2\" (UID: \"00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e\") " pod="openshift-image-registry/node-ca-sgfk2"
Jul 31 21:36:02.381843 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.381821    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwqm\" (UniqueName: \"kubernetes.io/projected/6d70e996-a47e-4761-b0c0-0795bc2bd2be-kube-api-access-whwqm\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:02.384779 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.384749    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xqj6\" (UniqueName: \"kubernetes.io/projected/927efd27-4c3e-40a8-ac7b-14322f6ec2e7-kube-api-access-6xqj6\") pod \"multus-mh7vc\" (UID: \"927efd27-4c3e-40a8-ac7b-14322f6ec2e7\") " pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.449073 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.449041    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jk845"
Jul 31 21:36:02.459281 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.459239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dptpt"
Jul 31 21:36:02.469358 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.469317    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:02.475527 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.475499    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6wcq8"
Jul 31 21:36:02.483582 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.483558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d2sf8"
Jul 31 21:36:02.491639 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.491609    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b"
Jul 31 21:36:02.500648 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.500619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sgfk2"
Jul 31 21:36:02.508645 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.508621    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mh7vc"
Jul 31 21:36:02.523493 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.523471    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:02.875429 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.875342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tvh\" (UniqueName: \"kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh\") pod \"network-check-target-lwcrg\" (UID: \"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e\") " pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:02.875429 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.875410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:02.875632 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:02.875452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:02.875632 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.875543    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 21:36:02.875632 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.875573    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 21:36:02.875632 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.875580    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:02.875818 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.875647    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs podName:6d70e996-a47e-4761-b0c0-0795bc2bd2be nodeName:}" failed. No retries permitted until 2026-07-31 21:36:03.875631318 +0000 UTC m=+4.138379661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs") pod "network-metrics-daemon-fpcs6" (UID: "6d70e996-a47e-4761-b0c0-0795bc2bd2be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:02.875818 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.875581    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:02.875818 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.875585    2561 projected.go:196] Error preparing data for projected volume kube-api-access-x2tvh for pod openshift-network-diagnostics/network-check-target-lwcrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:02.875818 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.875726    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret podName:2d760f73-be89-4905-88ed-68554584ed68 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:03.875709187 +0000 UTC m=+4.138457556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret") pod "global-pull-secret-syncer-rw7xc" (UID: "2d760f73-be89-4905-88ed-68554584ed68") : object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:02.875818 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:02.875794    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh podName:f626a3ca-8916-4bc6-8c74-71fa3b2cee8e nodeName:}" failed. No retries permitted until 2026-07-31 21:36:03.875766126 +0000 UTC m=+4.138514483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2tvh" (UniqueName: "kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh") pod "network-check-target-lwcrg" (UID: "f626a3ca-8916-4bc6-8c74-71fa3b2cee8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:03.214186 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:03.214093    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 21:31:02 +0000 UTC" deadline="2028-01-13 22:48:20.539374084 +0000 UTC"
Jul 31 21:36:03.214186 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:03.214126    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12745h12m17.325251746s"
Jul 31 21:36:03.537812 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:03.537775    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228725d4_468e_4b0c_b209_fc141baf250b.slice/crio-11f2727e6c2b76acacdf2f4a71cc52314f61e7b500a0b7e3bffe138a9d8dcda7 WatchSource:0}: Error finding container 11f2727e6c2b76acacdf2f4a71cc52314f61e7b500a0b7e3bffe138a9d8dcda7: Status 404 returned error can't find the container with id 11f2727e6c2b76acacdf2f4a71cc52314f61e7b500a0b7e3bffe138a9d8dcda7
Jul 31 21:36:03.539830 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:03.539718    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9feb5b8a_a382_4bb3_8a58_0f6fa2a0edcc.slice/crio-0229b450991ed614f118eaa41b51ee1fd31e7ff652bc04863be71b87f60db99a WatchSource:0}: Error finding container 0229b450991ed614f118eaa41b51ee1fd31e7ff652bc04863be71b87f60db99a: Status 404 returned error can't find the container with id 0229b450991ed614f118eaa41b51ee1fd31e7ff652bc04863be71b87f60db99a
Jul 31 21:36:03.542721 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:03.542592    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13d700b_a730_4cf1_b3c7_00f71811a639.slice/crio-77948aab78267c04977ff36ebded4da26f7b7b1b2894bf886f478acfe8ef34c5 WatchSource:0}: Error finding container 77948aab78267c04977ff36ebded4da26f7b7b1b2894bf886f478acfe8ef34c5: Status 404 returned error can't find the container with id 77948aab78267c04977ff36ebded4da26f7b7b1b2894bf886f478acfe8ef34c5
Jul 31 21:36:03.543800 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:03.543779    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc173f8c2_063d_45d3_9a97_f83842088f2b.slice/crio-004e092c79a0f0c04fb2974a1173bfa685d45633a3cc005ac1db1ed067a26d5b WatchSource:0}: Error finding container 004e092c79a0f0c04fb2974a1173bfa685d45633a3cc005ac1db1ed067a26d5b: Status 404 returned error can't find the container with id 004e092c79a0f0c04fb2974a1173bfa685d45633a3cc005ac1db1ed067a26d5b
Jul 31 21:36:03.545628 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:03.545604    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb35c507_56b4_4340_b132_bb962698c791.slice/crio-3cf95cd2b7ca87c11ea7ce845d436d81f3fee2ab12c810c59a03f741bf2f63ba WatchSource:0}: Error finding container 3cf95cd2b7ca87c11ea7ce845d436d81f3fee2ab12c810c59a03f741bf2f63ba: Status 404 returned error can't find the container with id 3cf95cd2b7ca87c11ea7ce845d436d81f3fee2ab12c810c59a03f741bf2f63ba
Jul 31 21:36:03.546555 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:03.546524    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927efd27_4c3e_40a8_ac7b_14322f6ec2e7.slice/crio-0bef9409de748b1cdf2b02a85dcb16ea99e9dbede1090f6d9958a32f72ec646d WatchSource:0}: Error finding container 0bef9409de748b1cdf2b02a85dcb16ea99e9dbede1090f6d9958a32f72ec646d: Status 404 returned error can't find the container with id 0bef9409de748b1cdf2b02a85dcb16ea99e9dbede1090f6d9958a32f72ec646d
Jul 31 21:36:03.548568 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:03.548542    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6f7a4b_33a6_4f10_8e3d_c70fa4e0119d.slice/crio-9980c7da99db1fd928fa47877ce82e899e72cd901b8b1871ceb8f06956096ba4 WatchSource:0}: Error finding container 9980c7da99db1fd928fa47877ce82e899e72cd901b8b1871ceb8f06956096ba4: Status 404 returned error can't find the container with id 9980c7da99db1fd928fa47877ce82e899e72cd901b8b1871ceb8f06956096ba4
Jul 31 21:36:03.550570 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:03.550405    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3166d1ee_0352_46a8_a585_0189cba32bd6.slice/crio-9596ca865d76b1c840e4f4d664252aee9ef07816397b1d8413ad8819010ce178 WatchSource:0}: Error finding container 9596ca865d76b1c840e4f4d664252aee9ef07816397b1d8413ad8819010ce178: Status 404 returned error can't find the container with id 9596ca865d76b1c840e4f4d664252aee9ef07816397b1d8413ad8819010ce178
Jul 31 21:36:03.551515 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:03.551484    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bd2ff5_1c1e_4ba2_8a4d_353cab6f3d3e.slice/crio-4d32ea68a1892fb835eb21cfe822b19cdb260f2a5a9befa2a659965661c5809a WatchSource:0}: Error finding container 4d32ea68a1892fb835eb21cfe822b19cdb260f2a5a9befa2a659965661c5809a: Status 404 returned error can't find the container with id 4d32ea68a1892fb835eb21cfe822b19cdb260f2a5a9befa2a659965661c5809a
Jul 31 21:36:03.884050 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:03.883871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tvh\" (UniqueName: \"kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh\") pod \"network-check-target-lwcrg\" (UID: \"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e\") " pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:03.884178 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:03.884076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:03.884178 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:03.884100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:03.884178 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:03.884023    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 21:36:03.884178 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:03.884163    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 21:36:03.884178 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:03.884178    2561 projected.go:196] Error preparing data for projected volume kube-api-access-x2tvh for pod openshift-network-diagnostics/network-check-target-lwcrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:03.884409 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:03.884213    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:03.884409 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:03.884206    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:03.884409 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:03.884249    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh podName:f626a3ca-8916-4bc6-8c74-71fa3b2cee8e nodeName:}" failed. No retries permitted until 2026-07-31 21:36:05.884225463 +0000 UTC m=+6.146973813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2tvh" (UniqueName: "kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh") pod "network-check-target-lwcrg" (UID: "f626a3ca-8916-4bc6-8c74-71fa3b2cee8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:03.884409 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:03.884269    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret podName:2d760f73-be89-4905-88ed-68554584ed68 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:05.884260016 +0000 UTC m=+6.147008361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret") pod "global-pull-secret-syncer-rw7xc" (UID: "2d760f73-be89-4905-88ed-68554584ed68") : object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:03.884409 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:03.884286    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs podName:6d70e996-a47e-4761-b0c0-0795bc2bd2be nodeName:}" failed. No retries permitted until 2026-07-31 21:36:05.884277136 +0000 UTC m=+6.147025483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs") pod "network-metrics-daemon-fpcs6" (UID: "6d70e996-a47e-4761-b0c0-0795bc2bd2be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:04.214441 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.214273    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 21:31:02 +0000 UTC" deadline="2028-02-08 05:51:51.486223103 +0000 UTC"
Jul 31 21:36:04.214441 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.214341    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13352h15m47.271887069s"
Jul 31 21:36:04.224995 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.223906    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:04.224995 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:04.224032    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:04.224995 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.224519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:04.224995 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:04.224630    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:04.224995 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.224695    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:04.224995 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:04.224767    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:04.259684 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.259649    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nfmj9" event={"ID":"3166d1ee-0352-46a8-a585-0189cba32bd6","Type":"ContainerStarted","Data":"9596ca865d76b1c840e4f4d664252aee9ef07816397b1d8413ad8819010ce178"}
Jul 31 21:36:04.262786 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.262713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mh7vc" event={"ID":"927efd27-4c3e-40a8-ac7b-14322f6ec2e7","Type":"ContainerStarted","Data":"0bef9409de748b1cdf2b02a85dcb16ea99e9dbede1090f6d9958a32f72ec646d"}
Jul 31 21:36:04.269350 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.269248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj" event={"ID":"c173f8c2-063d-45d3-9a97-f83842088f2b","Type":"ContainerStarted","Data":"004e092c79a0f0c04fb2974a1173bfa685d45633a3cc005ac1db1ed067a26d5b"}
Jul 31 21:36:04.290563 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.290473    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b" event={"ID":"228725d4-468e-4b0c-b209-fc141baf250b","Type":"ContainerStarted","Data":"11f2727e6c2b76acacdf2f4a71cc52314f61e7b500a0b7e3bffe138a9d8dcda7"}
Jul 31 21:36:04.301046 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.300510    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-241.ec2.internal" event={"ID":"b97b9740959824207498b954ade89901","Type":"ContainerStarted","Data":"71cbdb47f23b9f7b722562775520413e29abbabe0c6ca96275b7d318e2096fa5"}
Jul 31 21:36:04.313291 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.313190    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d2sf8" event={"ID":"cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d","Type":"ContainerStarted","Data":"9980c7da99db1fd928fa47877ce82e899e72cd901b8b1871ceb8f06956096ba4"}
Jul 31 21:36:04.318156 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.318093    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-241.ec2.internal" podStartSLOduration=3.318072815 podStartE2EDuration="3.318072815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:04.316864763 +0000 UTC m=+4.579613166" watchObservedRunningTime="2026-07-31 21:36:04.318072815 +0000 UTC m=+4.580821183"
Jul 31 21:36:04.318635 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.318603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dptpt" event={"ID":"cb35c507-56b4-4340-b132-bb962698c791","Type":"ContainerStarted","Data":"3cf95cd2b7ca87c11ea7ce845d436d81f3fee2ab12c810c59a03f741bf2f63ba"}
Jul 31 21:36:04.336799 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.336722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6wcq8" event={"ID":"f13d700b-a730-4cf1-b3c7-00f71811a639","Type":"ContainerStarted","Data":"77948aab78267c04977ff36ebded4da26f7b7b1b2894bf886f478acfe8ef34c5"}
Jul 31 21:36:04.339881 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.339789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk845" event={"ID":"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc","Type":"ContainerStarted","Data":"0229b450991ed614f118eaa41b51ee1fd31e7ff652bc04863be71b87f60db99a"}
Jul 31 21:36:04.343861 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.342517    2561 generic.go:356] "Generic (PLEG): container finished" podID="74b4ba3bfec4eba5141be9bc0bf6ece8" containerID="e90636d34ce4f5751b60a6be190b106bb990e654241faf908700a713bd670cf1" exitCode=0
Jul 31 21:36:04.343861 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.342594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal" event={"ID":"74b4ba3bfec4eba5141be9bc0bf6ece8","Type":"ContainerDied","Data":"e90636d34ce4f5751b60a6be190b106bb990e654241faf908700a713bd670cf1"}
Jul 31 21:36:04.351118 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:04.351084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sgfk2" event={"ID":"00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e","Type":"ContainerStarted","Data":"4d32ea68a1892fb835eb21cfe822b19cdb260f2a5a9befa2a659965661c5809a"}
Jul 31 21:36:05.375350 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:05.375274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal" event={"ID":"74b4ba3bfec4eba5141be9bc0bf6ece8","Type":"ContainerStarted","Data":"759739380ca4444e3af4b9f0cb63f29c4284f1633c17d185a8d7e9e6ec50d436"}
Jul 31 21:36:05.902498 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:05.902457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tvh\" (UniqueName: \"kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh\") pod \"network-check-target-lwcrg\" (UID: \"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e\") " pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:05.902684 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:05.902528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:05.902684 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:05.902562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:05.902797 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:05.902690    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:05.902797 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:05.902750    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret podName:2d760f73-be89-4905-88ed-68554584ed68 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:09.902731984 +0000 UTC m=+10.165480334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret") pod "global-pull-secret-syncer-rw7xc" (UID: "2d760f73-be89-4905-88ed-68554584ed68") : object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:05.903180 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:05.903160    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 21:36:05.903266 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:05.903189    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 21:36:05.903266 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:05.903202    2561 projected.go:196] Error preparing data for projected volume kube-api-access-x2tvh for pod openshift-network-diagnostics/network-check-target-lwcrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:05.903266 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:05.903249    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh podName:f626a3ca-8916-4bc6-8c74-71fa3b2cee8e nodeName:}" failed. No retries permitted until 2026-07-31 21:36:09.903234284 +0000 UTC m=+10.165982628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2tvh" (UniqueName: "kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh") pod "network-check-target-lwcrg" (UID: "f626a3ca-8916-4bc6-8c74-71fa3b2cee8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:05.903443 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:05.903311    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:05.903443 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:05.903352    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs podName:6d70e996-a47e-4761-b0c0-0795bc2bd2be nodeName:}" failed. No retries permitted until 2026-07-31 21:36:09.903332562 +0000 UTC m=+10.166080911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs") pod "network-metrics-daemon-fpcs6" (UID: "6d70e996-a47e-4761-b0c0-0795bc2bd2be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:06.224876 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:06.224242    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:06.224876 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:06.224373    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:06.224876 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:06.224259    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:06.224876 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:06.224476    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:06.224876 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:06.224242    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:06.224876 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:06.224562    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:08.226551 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:08.226205    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:08.226551 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:08.226206    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:08.226551 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:08.226324    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:08.226551 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:08.226349    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:08.226551 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:08.226435    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:08.226551 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:08.226516    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:09.939161 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:09.938831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tvh\" (UniqueName: \"kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh\") pod \"network-check-target-lwcrg\" (UID: \"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e\") " pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:09.939161 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:09.938899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:09.939161 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:09.938932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:09.939161 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:09.939050    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:09.939161 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:09.939121    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret podName:2d760f73-be89-4905-88ed-68554584ed68 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:17.93909332 +0000 UTC m=+18.201841681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret") pod "global-pull-secret-syncer-rw7xc" (UID: "2d760f73-be89-4905-88ed-68554584ed68") : object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:09.939864 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:09.939553    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 21:36:09.939864 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:09.939572    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 21:36:09.939864 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:09.939585    2561 projected.go:196] Error preparing data for projected volume kube-api-access-x2tvh for pod openshift-network-diagnostics/network-check-target-lwcrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:09.939864 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:09.939627    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh podName:f626a3ca-8916-4bc6-8c74-71fa3b2cee8e nodeName:}" failed. No retries permitted until 2026-07-31 21:36:17.939613974 +0000 UTC m=+18.202362330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2tvh" (UniqueName: "kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh") pod "network-check-target-lwcrg" (UID: "f626a3ca-8916-4bc6-8c74-71fa3b2cee8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:09.939864 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:09.939687    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:09.939864 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:09.939721    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs podName:6d70e996-a47e-4761-b0c0-0795bc2bd2be nodeName:}" failed. No retries permitted until 2026-07-31 21:36:17.939710004 +0000 UTC m=+18.202458349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs") pod "network-metrics-daemon-fpcs6" (UID: "6d70e996-a47e-4761-b0c0-0795bc2bd2be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:10.222101 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:10.222018    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:10.222273 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:10.222109    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:10.222450 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:10.222431    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:10.222548 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:10.222525    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:10.222681 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:10.222623    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:10.222746 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:10.222710    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:12.224944 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:12.224915    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:12.225429 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:12.225039    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:12.225429 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:12.225422    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:12.225528 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:12.225510    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:12.225598 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:12.225556    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:12.225653 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:12.225633    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:14.221954 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:14.221915    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:14.221954 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:14.221915    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:14.222510 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:14.221915    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:14.222510 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:14.222050    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:14.222510 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:14.222177    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:14.222510 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:14.222245    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:16.221569 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:16.221537    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:16.221569 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:16.221567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:16.222112 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:16.221534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:16.222112 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:16.221668    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:16.222112 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:16.221743    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:16.222112 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:16.221800    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:18.005809 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:18.005767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tvh\" (UniqueName: \"kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh\") pod \"network-check-target-lwcrg\" (UID: \"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e\") " pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:18.006362 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:18.005818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:18.006362 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:18.005921    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:18.006362 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:18.005943    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 21:36:18.006362 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:18.005963    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 21:36:18.006362 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:18.005974    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs podName:6d70e996-a47e-4761-b0c0-0795bc2bd2be nodeName:}" failed. No retries permitted until 2026-07-31 21:36:34.005958601 +0000 UTC m=+34.268706946 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs") pod "network-metrics-daemon-fpcs6" (UID: "6d70e996-a47e-4761-b0c0-0795bc2bd2be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:18.006362 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:18.005975    2561 projected.go:196] Error preparing data for projected volume kube-api-access-x2tvh for pod openshift-network-diagnostics/network-check-target-lwcrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:18.006362 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:18.005955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:18.006362 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:18.006014    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh podName:f626a3ca-8916-4bc6-8c74-71fa3b2cee8e nodeName:}" failed. No retries permitted until 2026-07-31 21:36:34.006006668 +0000 UTC m=+34.268755049 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2tvh" (UniqueName: "kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh") pod "network-check-target-lwcrg" (UID: "f626a3ca-8916-4bc6-8c74-71fa3b2cee8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:18.006362 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:18.006041    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:18.006362 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:18.006095    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret podName:2d760f73-be89-4905-88ed-68554584ed68 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:34.006079622 +0000 UTC m=+34.268827964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret") pod "global-pull-secret-syncer-rw7xc" (UID: "2d760f73-be89-4905-88ed-68554584ed68") : object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:18.221896 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:18.221873    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:18.221997 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:18.221886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:18.222033 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:18.221988    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:18.222071 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:18.222047    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:18.222106 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:18.222062    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:18.222184 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:18.222166    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:20.222334 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:20.222292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:20.222808 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:20.222396    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:20.222808 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:20.222453    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:20.222808 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:20.222577    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:20.222808 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:20.222591    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:20.222808 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:20.222679    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:22.222357 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.222052    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:22.223126 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.222049    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:22.223126 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:22.222469    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:22.223126 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:22.222494    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:22.223126 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.222090    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:22.223126 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:22.222558    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:22.418815 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.418780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mh7vc" event={"ID":"927efd27-4c3e-40a8-ac7b-14322f6ec2e7","Type":"ContainerStarted","Data":"4b8f05aad0c5e0259753521fc61a6c9a6f02456f3199ccbe95b7714b3f1a777d"}
Jul 31 21:36:22.421227 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.421205    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj" event={"ID":"c173f8c2-063d-45d3-9a97-f83842088f2b","Type":"ContainerStarted","Data":"f71472c03bd27afe31117ade680896687db698a79457a523a36c628913b0ae2c"}
Jul 31 21:36:22.421316 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.421260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj" event={"ID":"c173f8c2-063d-45d3-9a97-f83842088f2b","Type":"ContainerStarted","Data":"c2b92a903dab537589ace8ef1503d73898b62374939ecc75e5c9cab43e39a77e"}
Jul 31 21:36:22.421316 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.421295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj" event={"ID":"c173f8c2-063d-45d3-9a97-f83842088f2b","Type":"ContainerStarted","Data":"b03e07e1c86a932d02e00fdc166ad9b422dde0d868f4e3aa66748ee7d4c497df"}
Jul 31 21:36:22.421410 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.421331    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj" event={"ID":"c173f8c2-063d-45d3-9a97-f83842088f2b","Type":"ContainerStarted","Data":"244fcd80ec050288623bdd02226d7b21241d6ac7423ba7ede2c33d60619b45e7"}
Jul 31 21:36:22.422669 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.422650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b" event={"ID":"228725d4-468e-4b0c-b209-fc141baf250b","Type":"ContainerStarted","Data":"db82de3c05c20358cfb44162a9479d669324a1afb2b2e8261850172f4f20974a"}
Jul 31 21:36:22.423930 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.423906    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d2sf8" event={"ID":"cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d","Type":"ContainerStarted","Data":"d0f530bd6351309330436cdbbbfd4c7f8583ee6c93d420e1c93afac064982b90"}
Jul 31 21:36:22.425088 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.425072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6wcq8" event={"ID":"f13d700b-a730-4cf1-b3c7-00f71811a639","Type":"ContainerStarted","Data":"2ddeb53de504cbe629ade892604c828970cb39c86125e7e46772a2f4d0367db0"}
Jul 31 21:36:22.426396 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.426377    2561 generic.go:356] "Generic (PLEG): container finished" podID="9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc" containerID="c9b5c62cc1905820505f48fd76bbe0f6c30e1c3896d802c2b3a40df895df3b17" exitCode=0
Jul 31 21:36:22.426482 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.426441    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk845" event={"ID":"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc","Type":"ContainerDied","Data":"c9b5c62cc1905820505f48fd76bbe0f6c30e1c3896d802c2b3a40df895df3b17"}
Jul 31 21:36:22.427810 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.427793    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sgfk2" event={"ID":"00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e","Type":"ContainerStarted","Data":"9dea5729ec85d37faa6424d15546d9c812b23a35c0d14283839b32a2ee0b4750"}
Jul 31 21:36:22.429258 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.429238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nfmj9" event={"ID":"3166d1ee-0352-46a8-a585-0189cba32bd6","Type":"ContainerStarted","Data":"d16252fcdc2dd5e30d851f70cc375d86aebf4459fdc760caba5e2ec13ace6ee6"}
Jul 31 21:36:22.436163 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.436112    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-241.ec2.internal" podStartSLOduration=21.436101176 podStartE2EDuration="21.436101176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:05.40035741 +0000 UTC m=+5.663105780" watchObservedRunningTime="2026-07-31 21:36:22.436101176 +0000 UTC m=+22.698849539"
Jul 31 21:36:22.436505 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.436484    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mh7vc" podStartSLOduration=3.044472748 podStartE2EDuration="21.436478681s" totalImagesPullingTime="18.392005933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:22.435566162 +0000 UTC m=+22.698314525" watchObservedRunningTime="2026-07-31 21:36:22.436478681 +0000 UTC m=+22.699227044"
Jul 31 21:36:22.474266 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.474200    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sgfk2" podStartSLOduration=3.14939336 podStartE2EDuration="21.474185597s" totalImagesPullingTime="18.324792237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:22.4738178 +0000 UTC m=+22.736566164" watchObservedRunningTime="2026-07-31 21:36:22.474185597 +0000 UTC m=+22.736933965"
Jul 31 21:36:22.488428 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.488388    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nfmj9" podStartSLOduration=6.823360261 podStartE2EDuration="21.488370972s" totalImagesPullingTime="14.665010711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:22.487927001 +0000 UTC m=+22.750675361" watchObservedRunningTime="2026-07-31 21:36:22.488370972 +0000 UTC m=+22.751119337"
Jul 31 21:36:22.505487 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.505447    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6wcq8" podStartSLOduration=4.154531708 podStartE2EDuration="22.505428868s" totalImagesPullingTime="18.35089716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:22.505189245 +0000 UTC m=+22.767937610" watchObservedRunningTime="2026-07-31 21:36:22.505428868 +0000 UTC m=+22.768177232"
Jul 31 21:36:22.525054 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:22.525007    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d2sf8" podStartSLOduration=4.194141241 podStartE2EDuration="22.524990601s" totalImagesPullingTime="18.33084936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:22.524962163 +0000 UTC m=+22.787710534" watchObservedRunningTime="2026-07-31 21:36:22.524990601 +0000 UTC m=+22.787738962"
Jul 31 21:36:23.418353 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:23.418315    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 21:36:23.435377 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:23.435339    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj" event={"ID":"c173f8c2-063d-45d3-9a97-f83842088f2b","Type":"ContainerStarted","Data":"f95434057b0e6d973ad3f22002dd555063b96c768939976c6fd319544ca633b8"}
Jul 31 21:36:23.435494 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:23.435461    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj" event={"ID":"c173f8c2-063d-45d3-9a97-f83842088f2b","Type":"ContainerStarted","Data":"fec2153b5faf741a1da6db396c197e6c6636060c3d9a856d11edff2975424b83"}
Jul 31 21:36:23.437372 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:23.437340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b" event={"ID":"228725d4-468e-4b0c-b209-fc141baf250b","Type":"ContainerStarted","Data":"ccb4a11b119443b9fd85b7d176fde56bb2accff8d75f6de2a5eb761779c11c93"}
Jul 31 21:36:23.439003 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:23.438922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dptpt" event={"ID":"cb35c507-56b4-4340-b132-bb962698c791","Type":"ContainerStarted","Data":"28e35a287f2a5ad61969ab9f4b9abd206c74a4a52a4846fcc1759f61fdbf99c3"}
Jul 31 21:36:23.453938 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:23.453872    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dptpt" podStartSLOduration=8.783886201 podStartE2EDuration="23.453856671s" totalImagesPullingTime="14.66997047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:23.453742816 +0000 UTC m=+23.716491181" watchObservedRunningTime="2026-07-31 21:36:23.453856671 +0000 UTC m=+23.716605067"
Jul 31 21:36:24.221676 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:24.221639    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:24.222030 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:24.221639    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:24.222030 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:24.221771    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:24.222030 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:24.221814    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:24.222030 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:24.221639    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:24.222030 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:24.221895    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:24.225739 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:24.225672    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T21:36:23.418336147Z","UUID":"ebe1d443-daa6-44c9-813b-dbe5bdf0e90f","Handler":null,"Name":"","Endpoint":""}
Jul 31 21:36:24.227414 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:24.227392    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 21:36:24.227518 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:24.227436    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 21:36:24.561999 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:24.561964    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:24.562629 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:24.562611    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:25.448417 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:25.448167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj" event={"ID":"c173f8c2-063d-45d3-9a97-f83842088f2b","Type":"ContainerStarted","Data":"9a321e4c38869c29235f9cab2c5f142ba7f405d06d1fc15573330adcf8b03498"}
Jul 31 21:36:25.450526 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:25.450492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b" event={"ID":"228725d4-468e-4b0c-b209-fc141baf250b","Type":"ContainerStarted","Data":"2af41bc23e736167344816def9247185a35cf1f4e49ce04dd5f8b52bfe840041"}
Jul 31 21:36:25.450812 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:25.450782    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:25.451313 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:25.451294    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nfmj9"
Jul 31 21:36:25.470310 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:25.470263    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjd8b" podStartSLOduration=3.654275773 podStartE2EDuration="24.470246851s" totalImagesPullingTime="20.815971078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:25.469477343 +0000 UTC m=+25.732225706" watchObservedRunningTime="2026-07-31 21:36:25.470246851 +0000 UTC m=+25.732995215"
Jul 31 21:36:26.221403 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:26.221372    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:26.221844 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:26.221372    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:26.221844 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:26.221504    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:26.221844 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:26.221565    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:26.221844 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:26.221577    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:26.221844 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:26.221659    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:27.458359 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:27.458011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj" event={"ID":"c173f8c2-063d-45d3-9a97-f83842088f2b","Type":"ContainerStarted","Data":"3fc6b99981786ef7bc0038347c3d52fb97c83982c732dcbcf69355f1118c1434"}
Jul 31 21:36:27.459115 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:27.458390    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:27.460018 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:27.459987    2561 generic.go:356] "Generic (PLEG): container finished" podID="9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc" containerID="7dd5e551e26824d3966f85cf38f60f905cd8f908e439e5c78e7b3aacd4fdeb45" exitCode=0
Jul 31 21:36:27.460115 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:27.460084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk845" event={"ID":"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc","Type":"ContainerDied","Data":"7dd5e551e26824d3966f85cf38f60f905cd8f908e439e5c78e7b3aacd4fdeb45"}
Jul 31 21:36:27.474068 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:27.474048    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:27.486104 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:27.486063    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj" podStartSLOduration=9.090556868 podStartE2EDuration="27.486029606s" totalImagesPullingTime="18.395472738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:27.485635751 +0000 UTC m=+27.748384132" watchObservedRunningTime="2026-07-31 21:36:27.486029606 +0000 UTC m=+27.748777970"
Jul 31 21:36:28.221996 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:28.221953    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:28.222246 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:28.221954    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:28.222246 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:28.222079    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:28.222246 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:28.222091    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:28.222425 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:28.222266    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:28.222425 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:28.222354    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:28.462368 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:28.462336    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:28.462821 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:28.462397    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:28.476598 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:28.476527    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:36:29.344504 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:29.344470    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rw7xc"]
Jul 31 21:36:29.344755 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:29.344624    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:29.344755 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:29.344715    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:29.348475 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:29.348451    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lwcrg"]
Jul 31 21:36:29.348602 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:29.348554    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:29.348679 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:29.348657    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:29.349033 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:29.349014    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fpcs6"]
Jul 31 21:36:29.349127 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:29.349115    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:29.349279 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:29.349258    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:29.466068 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:29.466034    2561 generic.go:356] "Generic (PLEG): container finished" podID="9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc" containerID="d17e42551c1988fc94f90013b104f90c9449887345be4cc09a288435021cdd42" exitCode=0
Jul 31 21:36:29.466503 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:29.466128    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk845" event={"ID":"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc","Type":"ContainerDied","Data":"d17e42551c1988fc94f90013b104f90c9449887345be4cc09a288435021cdd42"}
Jul 31 21:36:31.222365 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:31.222293    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:31.222760 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:31.222370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:31.222760 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:31.222466    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:31.222760 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:31.222517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:31.222760 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:31.222613    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:31.222760 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:31.222710    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:31.473180 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:31.473088    2561 generic.go:356] "Generic (PLEG): container finished" podID="9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc" containerID="c448587aaf8fd88c821ae1eed6009efa9076f0b1ae21f04edaa665be5161831f" exitCode=0
Jul 31 21:36:31.473180 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:31.473129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk845" event={"ID":"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc","Type":"ContainerDied","Data":"c448587aaf8fd88c821ae1eed6009efa9076f0b1ae21f04edaa665be5161831f"}
Jul 31 21:36:33.221976 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:33.221944    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:33.222467 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:33.221943    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:33.222467 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:33.222051    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw7xc" podUID="2d760f73-be89-4905-88ed-68554584ed68"
Jul 31 21:36:33.222467 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:33.222177    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpcs6" podUID="6d70e996-a47e-4761-b0c0-0795bc2bd2be"
Jul 31 21:36:33.222467 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:33.221943    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:33.222467 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:33.222278    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwcrg" podUID="f626a3ca-8916-4bc6-8c74-71fa3b2cee8e"
Jul 31 21:36:34.032724 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.032688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:34.032911 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.032737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:34.032911 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.032790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tvh\" (UniqueName: \"kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh\") pod \"network-check-target-lwcrg\" (UID: \"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e\") " pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:34.032911 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.032863    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:34.033079 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.032915    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:34.033079 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.032934    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs podName:6d70e996-a47e-4761-b0c0-0795bc2bd2be nodeName:}" failed. No retries permitted until 2026-07-31 21:37:06.032912601 +0000 UTC m=+66.295660963 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs") pod "network-metrics-daemon-fpcs6" (UID: "6d70e996-a47e-4761-b0c0-0795bc2bd2be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 21:36:34.033079 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.032979    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret podName:2d760f73-be89-4905-88ed-68554584ed68 nodeName:}" failed. No retries permitted until 2026-07-31 21:37:06.032961842 +0000 UTC m=+66.295710415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret") pod "global-pull-secret-syncer-rw7xc" (UID: "2d760f73-be89-4905-88ed-68554584ed68") : object "kube-system"/"original-pull-secret" not registered
Jul 31 21:36:34.033079 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.032982    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 21:36:34.033079 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.033004    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 21:36:34.033079 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.033018    2561 projected.go:196] Error preparing data for projected volume kube-api-access-x2tvh for pod openshift-network-diagnostics/network-check-target-lwcrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:34.033331 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.033086    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh podName:f626a3ca-8916-4bc6-8c74-71fa3b2cee8e nodeName:}" failed. No retries permitted until 2026-07-31 21:37:06.033054695 +0000 UTC m=+66.295803042 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2tvh" (UniqueName: "kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh") pod "network-check-target-lwcrg" (UID: "f626a3ca-8916-4bc6-8c74-71fa3b2cee8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 21:36:34.553962 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.553934    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-241.ec2.internal" event="NodeReady"
Jul 31 21:36:34.554636 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.554073    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 21:36:34.592190 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.592157    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86b4d994cc-8wmt4"]
Jul 31 21:36:34.598056 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.598033    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.600005 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.599983    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 21:36:34.600005 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.599995    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 21:36:34.600465 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.600445    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 21:36:34.600862 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.600721    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dbtts\""
Jul 31 21:36:34.607583 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.607561    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 21:36:34.613694 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.613672    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xbrvr"]
Jul 31 21:36:34.617710 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.617690    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86b4d994cc-8wmt4"]
Jul 31 21:36:34.617810 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.617727    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r9jbm"]
Jul 31 21:36:34.617861 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.617829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.622066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.622048    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9jbm"
Jul 31 21:36:34.622455 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.622335    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 21:36:34.622557 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.622505    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jgwbq\""
Jul 31 21:36:34.622631 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.622581    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 21:36:34.623986 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.623805    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 21:36:34.623986 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.623844    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vzfj5\""
Jul 31 21:36:34.623986 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.623889    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 21:36:34.624475 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.624438    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 21:36:34.635050 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.634952    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9jbm"]
Jul 31 21:36:34.636522 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.636378    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xbrvr"]
Jul 31 21:36:34.738721 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.738628    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.738721 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.738675    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-installation-pull-secrets\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.738721 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.738702    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.738986 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.738754    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-certificates\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.738986 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.738775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7360b4d5-6dd2-467d-a3c8-75167012a942-config-volume\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.738986 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.738799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcrts\" (UniqueName: \"kubernetes.io/projected/7360b4d5-6dd2-467d-a3c8-75167012a942-kube-api-access-pcrts\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.738986 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.738826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-image-registry-private-configuration\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.738986 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.738852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c8044c4-cc78-4d68-ac3a-888285c681d1-ca-trust-extracted\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.738986 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.738877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n49df\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-kube-api-access-n49df\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.738986 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.738911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-bound-sa-token\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.738986 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.738957    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7360b4d5-6dd2-467d-a3c8-75167012a942-tmp-dir\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.739370 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.739180    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-trusted-ca\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.739370 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.739212    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ce0cdef-4292-4fa8-b7ff-c7fba201bbae-cert\") pod \"ingress-canary-r9jbm\" (UID: \"7ce0cdef-4292-4fa8-b7ff-c7fba201bbae\") " pod="openshift-ingress-canary/ingress-canary-r9jbm"
Jul 31 21:36:34.739370 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.739244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8vzn\" (UniqueName: \"kubernetes.io/projected/7ce0cdef-4292-4fa8-b7ff-c7fba201bbae-kube-api-access-q8vzn\") pod \"ingress-canary-r9jbm\" (UID: \"7ce0cdef-4292-4fa8-b7ff-c7fba201bbae\") " pod="openshift-ingress-canary/ingress-canary-r9jbm"
Jul 31 21:36:34.839610 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-trusted-ca\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.839798 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ce0cdef-4292-4fa8-b7ff-c7fba201bbae-cert\") pod \"ingress-canary-r9jbm\" (UID: \"7ce0cdef-4292-4fa8-b7ff-c7fba201bbae\") " pod="openshift-ingress-canary/ingress-canary-r9jbm"
Jul 31 21:36:34.839798 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8vzn\" (UniqueName: \"kubernetes.io/projected/7ce0cdef-4292-4fa8-b7ff-c7fba201bbae-kube-api-access-q8vzn\") pod \"ingress-canary-r9jbm\" (UID: \"7ce0cdef-4292-4fa8-b7ff-c7fba201bbae\") " pod="openshift-ingress-canary/ingress-canary-r9jbm"
Jul 31 21:36:34.839798 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.839798 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-installation-pull-secrets\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.839798 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.839798 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-certificates\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.839798 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7360b4d5-6dd2-467d-a3c8-75167012a942-config-volume\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.839798 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcrts\" (UniqueName: \"kubernetes.io/projected/7360b4d5-6dd2-467d-a3c8-75167012a942-kube-api-access-pcrts\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.840121 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-image-registry-private-configuration\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.840121 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c8044c4-cc78-4d68-ac3a-888285c681d1-ca-trust-extracted\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.840121 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n49df\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-kube-api-access-n49df\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.840121 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-bound-sa-token\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.840121 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.839887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7360b4d5-6dd2-467d-a3c8-75167012a942-tmp-dir\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.840366 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.840255    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7360b4d5-6dd2-467d-a3c8-75167012a942-tmp-dir\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.840603 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.840568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c8044c4-cc78-4d68-ac3a-888285c681d1-ca-trust-extracted\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.840700 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.840598    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-trusted-ca\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.840700 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.840580    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 21:36:34.840700 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.840641    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 21:36:34.840700 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.840656    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86b4d994cc-8wmt4: secret "image-registry-tls" not found
Jul 31 21:36:34.840700 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.840674    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls podName:7360b4d5-6dd2-467d-a3c8-75167012a942 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:35.340652828 +0000 UTC m=+35.603401171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls") pod "dns-default-xbrvr" (UID: "7360b4d5-6dd2-467d-a3c8-75167012a942") : secret "dns-default-metrics-tls" not found
Jul 31 21:36:34.840930 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:34.840707    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls podName:8c8044c4-cc78-4d68-ac3a-888285c681d1 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:35.340691204 +0000 UTC m=+35.603439568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls") pod "image-registry-86b4d994cc-8wmt4" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1") : secret "image-registry-tls" not found
Jul 31 21:36:34.841216 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.841193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-certificates\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.841357 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.841339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7360b4d5-6dd2-467d-a3c8-75167012a942-config-volume\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.844688 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.844626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-installation-pull-secrets\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.844688 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.844641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ce0cdef-4292-4fa8-b7ff-c7fba201bbae-cert\") pod \"ingress-canary-r9jbm\" (UID: \"7ce0cdef-4292-4fa8-b7ff-c7fba201bbae\") " pod="openshift-ingress-canary/ingress-canary-r9jbm"
Jul 31 21:36:34.844688 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.844624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-image-registry-private-configuration\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.852443 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.852416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-bound-sa-token\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.852711 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.852688    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n49df\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-kube-api-access-n49df\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:34.853503 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.853482    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcrts\" (UniqueName: \"kubernetes.io/projected/7360b4d5-6dd2-467d-a3c8-75167012a942-kube-api-access-pcrts\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:34.853989 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.853973    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8vzn\" (UniqueName: \"kubernetes.io/projected/7ce0cdef-4292-4fa8-b7ff-c7fba201bbae-kube-api-access-q8vzn\") pod \"ingress-canary-r9jbm\" (UID: \"7ce0cdef-4292-4fa8-b7ff-c7fba201bbae\") " pod="openshift-ingress-canary/ingress-canary-r9jbm"
Jul 31 21:36:34.937044 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:34.937009    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9jbm"
Jul 31 21:36:35.221744 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.221710    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:36:35.221924 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.221756    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:36:35.223015 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.222178    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:36:35.228033 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.228010    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 21:36:35.228177 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.228074    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2r85f\""
Jul 31 21:36:35.228436 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.228411    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 21:36:35.228436 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.228430    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 21:36:35.228629 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.228416    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 21:36:35.230320 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.230299    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zxl4n\""
Jul 31 21:36:35.343691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.343658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:35.343691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.343694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:35.343941 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:35.343821    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 21:36:35.343941 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:35.343837    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86b4d994cc-8wmt4: secret "image-registry-tls" not found
Jul 31 21:36:35.343941 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:35.343841    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 21:36:35.343941 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:35.343904    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls podName:8c8044c4-cc78-4d68-ac3a-888285c681d1 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:36.343883947 +0000 UTC m=+36.606632313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls") pod "image-registry-86b4d994cc-8wmt4" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1") : secret "image-registry-tls" not found
Jul 31 21:36:35.343941 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:35.343924    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls podName:7360b4d5-6dd2-467d-a3c8-75167012a942 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:36.343914869 +0000 UTC m=+36.606663226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls") pod "dns-default-xbrvr" (UID: "7360b4d5-6dd2-467d-a3c8-75167012a942") : secret "dns-default-metrics-tls" not found
Jul 31 21:36:35.753071 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:35.753043    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9jbm"]
Jul 31 21:36:35.756105 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:35.756079    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce0cdef_4292_4fa8_b7ff_c7fba201bbae.slice/crio-84a3fc9e3b7d4af800e87777c1325b060b59beab8d66dd9f34c65ee4dd985c1f WatchSource:0}: Error finding container 84a3fc9e3b7d4af800e87777c1325b060b59beab8d66dd9f34c65ee4dd985c1f: Status 404 returned error can't find the container with id 84a3fc9e3b7d4af800e87777c1325b060b59beab8d66dd9f34c65ee4dd985c1f
Jul 31 21:36:36.353125 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:36.353091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:36.353305 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:36.353145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:36.353305 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:36.353274    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 21:36:36.353305 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:36.353288    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 21:36:36.353305 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:36.353298    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86b4d994cc-8wmt4: secret "image-registry-tls" not found
Jul 31 21:36:36.353470 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:36.353356    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls podName:7360b4d5-6dd2-467d-a3c8-75167012a942 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:38.35333659 +0000 UTC m=+38.616084934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls") pod "dns-default-xbrvr" (UID: "7360b4d5-6dd2-467d-a3c8-75167012a942") : secret "dns-default-metrics-tls" not found
Jul 31 21:36:36.353470 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:36.353373    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls podName:8c8044c4-cc78-4d68-ac3a-888285c681d1 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:38.353364755 +0000 UTC m=+38.616113098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls") pod "image-registry-86b4d994cc-8wmt4" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1") : secret "image-registry-tls" not found
Jul 31 21:36:36.488410 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:36.488374    2561 generic.go:356] "Generic (PLEG): container finished" podID="9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc" containerID="fc00328ad893f6e8ff69c5bf4f06d0c156c6ef032f30f786c405ab38b1f9987e" exitCode=0
Jul 31 21:36:36.488573 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:36.488456    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk845" event={"ID":"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc","Type":"ContainerDied","Data":"fc00328ad893f6e8ff69c5bf4f06d0c156c6ef032f30f786c405ab38b1f9987e"}
Jul 31 21:36:36.489904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:36.489874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9jbm" event={"ID":"7ce0cdef-4292-4fa8-b7ff-c7fba201bbae","Type":"ContainerStarted","Data":"84a3fc9e3b7d4af800e87777c1325b060b59beab8d66dd9f34c65ee4dd985c1f"}
Jul 31 21:36:37.496713 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:37.496668    2561 generic.go:356] "Generic (PLEG): container finished" podID="9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc" containerID="10849601681b0db26ddffbe97d363412a8340fa8fa8203897922f5de6de1ab0f" exitCode=0
Jul 31 21:36:37.497270 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:37.496732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk845" event={"ID":"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc","Type":"ContainerDied","Data":"10849601681b0db26ddffbe97d363412a8340fa8fa8203897922f5de6de1ab0f"}
Jul 31 21:36:38.365948 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:38.365770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:38.366112 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:38.365960    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:38.366112 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:38.365911    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 21:36:38.366112 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:38.366031    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86b4d994cc-8wmt4: secret "image-registry-tls" not found
Jul 31 21:36:38.366112 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:38.366089    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls podName:8c8044c4-cc78-4d68-ac3a-888285c681d1 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:42.366071682 +0000 UTC m=+42.628820024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls") pod "image-registry-86b4d994cc-8wmt4" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1") : secret "image-registry-tls" not found
Jul 31 21:36:38.366112 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:38.366092    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 21:36:38.366299 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:38.366168    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls podName:7360b4d5-6dd2-467d-a3c8-75167012a942 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:42.36612921 +0000 UTC m=+42.628877552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls") pod "dns-default-xbrvr" (UID: "7360b4d5-6dd2-467d-a3c8-75167012a942") : secret "dns-default-metrics-tls" not found
Jul 31 21:36:38.502946 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:38.502865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk845" event={"ID":"9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc","Type":"ContainerStarted","Data":"6561b03e6329d4eb889e67540f6241dabdc65d54b872653435befaaffc73bf8e"}
Jul 31 21:36:38.504384 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:38.504360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9jbm" event={"ID":"7ce0cdef-4292-4fa8-b7ff-c7fba201bbae","Type":"ContainerStarted","Data":"f10535a773d97e3321fe7a28a6ab1172322e754ec6e9850d0960a1074dde28ac"}
Jul 31 21:36:38.526928 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:38.526885    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jk845" podStartSLOduration=9.033642774 podStartE2EDuration="38.526871422s" totalImagesPullingTime="29.493228648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:38.526300171 +0000 UTC m=+38.789048534" watchObservedRunningTime="2026-07-31 21:36:38.526871422 +0000 UTC m=+38.789619787"
Jul 31 21:36:38.542496 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:38.542445    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r9jbm" podStartSLOduration=2.201307668 podStartE2EDuration="4.542427167s" totalImagesPullingTime="2.341119499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:38.542072922 +0000 UTC m=+38.804821285" watchObservedRunningTime="2026-07-31 21:36:38.542427167 +0000 UTC m=+38.805175532"
Jul 31 21:36:41.517370 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.517336    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-dl4sm"]
Jul 31 21:36:41.547022 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.546995    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"]
Jul 31 21:36:41.547186 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.547123    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-dl4sm"
Jul 31 21:36:41.549254 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.549231    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7x6n2\""
Jul 31 21:36:41.570747 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.570722    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-dl4sm"]
Jul 31 21:36:41.570859 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.570755    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"]
Jul 31 21:36:41.570859 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.570829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:41.572688 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.572661    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 21:36:41.572793 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.572690    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rc6mk\""
Jul 31 21:36:41.573328 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.573310    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 31 21:36:41.573977 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.573961    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 31 21:36:41.574060 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.573966    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 21:36:41.588204 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.588177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxch\" (UniqueName: \"kubernetes.io/projected/24c3514d-5367-43f1-aa6e-9afb94a11da7-kube-api-access-fxxch\") pod \"network-check-source-65bdbbff45-dl4sm\" (UID: \"24c3514d-5367-43f1-aa6e-9afb94a11da7\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-dl4sm"
Jul 31 21:36:41.588302 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.588236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ce19efed-c49f-4620-abc9-6f312195bff0-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:41.588302 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.588258    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:41.588302 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.588296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdx7m\" (UniqueName: \"kubernetes.io/projected/ce19efed-c49f-4620-abc9-6f312195bff0-kube-api-access-sdx7m\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:41.624734 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.624704    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"]
Jul 31 21:36:41.648668 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.648644    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"]
Jul 31 21:36:41.648668 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.648674    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-4wqtz"]
Jul 31 21:36:41.648857 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.648774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"
Jul 31 21:36:41.651219 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.651190    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 31 21:36:41.651352 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.651273    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 31 21:36:41.651352 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.651308    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 31 21:36:41.651352 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.651313    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 31 21:36:41.651352 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.651323    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wppff\""
Jul 31 21:36:41.672736 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.672712    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-4wqtz"]
Jul 31 21:36:41.672852 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.672802    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.674818 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.674797    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 31 21:36:41.674936 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.674914    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-29dcd\""
Jul 31 21:36:41.675045 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.675001    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 31 21:36:41.675045 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.675024    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 31 21:36:41.675241 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.675075    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 31 21:36:41.688794 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.688773    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26a15230-5d14-464e-bc26-9df69a8d5967-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v7rql\" (UID: \"26a15230-5d14-464e-bc26-9df69a8d5967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"
Jul 31 21:36:41.688956 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.688804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxch\" (UniqueName: \"kubernetes.io/projected/24c3514d-5367-43f1-aa6e-9afb94a11da7-kube-api-access-fxxch\") pod \"network-check-source-65bdbbff45-dl4sm\" (UID: \"24c3514d-5367-43f1-aa6e-9afb94a11da7\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-dl4sm"
Jul 31 21:36:41.688956 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.688825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12a7ea88-8e73-44af-982d-adc960254ccc-trusted-ca\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.688956 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.688928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12a7ea88-8e73-44af-982d-adc960254ccc-serving-cert\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.689108 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.688954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwljn\" (UniqueName: \"kubernetes.io/projected/12a7ea88-8e73-44af-982d-adc960254ccc-kube-api-access-qwljn\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.689108 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.688982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ce19efed-c49f-4620-abc9-6f312195bff0-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:41.689108 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.689002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:41.689108 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.689050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26a15230-5d14-464e-bc26-9df69a8d5967-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v7rql\" (UID: \"26a15230-5d14-464e-bc26-9df69a8d5967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"
Jul 31 21:36:41.689108 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:41.689068    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 21:36:41.689108 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.689079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdx7m\" (UniqueName: \"kubernetes.io/projected/ce19efed-c49f-4620-abc9-6f312195bff0-kube-api-access-sdx7m\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:41.689108 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.689098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bj8m\" (UniqueName: \"kubernetes.io/projected/26a15230-5d14-464e-bc26-9df69a8d5967-kube-api-access-2bj8m\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v7rql\" (UID: \"26a15230-5d14-464e-bc26-9df69a8d5967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"
Jul 31 21:36:41.689108 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:41.689115    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls podName:ce19efed-c49f-4620-abc9-6f312195bff0 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:42.189102168 +0000 UTC m=+42.451850511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-8p8cq" (UID: "ce19efed-c49f-4620-abc9-6f312195bff0") : secret "cluster-monitoring-operator-tls" not found
Jul 31 21:36:41.689475 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.689162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12a7ea88-8e73-44af-982d-adc960254ccc-config\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.689947 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.689926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ce19efed-c49f-4620-abc9-6f312195bff0-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:41.694505 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.694490    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 31 21:36:41.704423 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.704403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxch\" (UniqueName: \"kubernetes.io/projected/24c3514d-5367-43f1-aa6e-9afb94a11da7-kube-api-access-fxxch\") pod \"network-check-source-65bdbbff45-dl4sm\" (UID: \"24c3514d-5367-43f1-aa6e-9afb94a11da7\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-dl4sm"
Jul 31 21:36:41.704499 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.704461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdx7m\" (UniqueName: \"kubernetes.io/projected/ce19efed-c49f-4620-abc9-6f312195bff0-kube-api-access-sdx7m\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:41.790003 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.789971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26a15230-5d14-464e-bc26-9df69a8d5967-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v7rql\" (UID: \"26a15230-5d14-464e-bc26-9df69a8d5967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"
Jul 31 21:36:41.790171 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.790012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bj8m\" (UniqueName: \"kubernetes.io/projected/26a15230-5d14-464e-bc26-9df69a8d5967-kube-api-access-2bj8m\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v7rql\" (UID: \"26a15230-5d14-464e-bc26-9df69a8d5967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"
Jul 31 21:36:41.790171 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.790035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12a7ea88-8e73-44af-982d-adc960254ccc-config\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.790295 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.790192    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26a15230-5d14-464e-bc26-9df69a8d5967-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v7rql\" (UID: \"26a15230-5d14-464e-bc26-9df69a8d5967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"
Jul 31 21:36:41.790295 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.790239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12a7ea88-8e73-44af-982d-adc960254ccc-trusted-ca\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.790387 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.790369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12a7ea88-8e73-44af-982d-adc960254ccc-serving-cert\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.790436 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.790408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwljn\" (UniqueName: \"kubernetes.io/projected/12a7ea88-8e73-44af-982d-adc960254ccc-kube-api-access-qwljn\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.790774 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.790729    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26a15230-5d14-464e-bc26-9df69a8d5967-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v7rql\" (UID: \"26a15230-5d14-464e-bc26-9df69a8d5967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"
Jul 31 21:36:41.790920 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.790888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12a7ea88-8e73-44af-982d-adc960254ccc-config\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.791518 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.791500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12a7ea88-8e73-44af-982d-adc960254ccc-trusted-ca\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.792554 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.792536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26a15230-5d14-464e-bc26-9df69a8d5967-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v7rql\" (UID: \"26a15230-5d14-464e-bc26-9df69a8d5967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"
Jul 31 21:36:41.792728 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.792708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12a7ea88-8e73-44af-982d-adc960254ccc-serving-cert\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.797869 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.797846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bj8m\" (UniqueName: \"kubernetes.io/projected/26a15230-5d14-464e-bc26-9df69a8d5967-kube-api-access-2bj8m\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v7rql\" (UID: \"26a15230-5d14-464e-bc26-9df69a8d5967\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"
Jul 31 21:36:41.798012 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.797992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwljn\" (UniqueName: \"kubernetes.io/projected/12a7ea88-8e73-44af-982d-adc960254ccc-kube-api-access-qwljn\") pod \"console-operator-5ccd6b445f-4wqtz\" (UID: \"12a7ea88-8e73-44af-982d-adc960254ccc\") " pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:41.856420 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.856384    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-dl4sm"
Jul 31 21:36:41.959065 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.959024    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"
Jul 31 21:36:41.981643 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:41.981613    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:42.003321 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:42.003222    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-dl4sm"]
Jul 31 21:36:42.007644 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:42.007615    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c3514d_5367_43f1_aa6e_9afb94a11da7.slice/crio-803b4f711a2444bd523b97cb41d53fbe77fc5efc3a1957f274845621cef2c136 WatchSource:0}: Error finding container 803b4f711a2444bd523b97cb41d53fbe77fc5efc3a1957f274845621cef2c136: Status 404 returned error can't find the container with id 803b4f711a2444bd523b97cb41d53fbe77fc5efc3a1957f274845621cef2c136
Jul 31 21:36:42.111322 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:42.111295    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql"]
Jul 31 21:36:42.114046 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:42.114021    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a15230_5d14_464e_bc26_9df69a8d5967.slice/crio-90afaf938c9b7a18558ccddb47b24a97541b3e712c32a92fd8492d597ed7e5ab WatchSource:0}: Error finding container 90afaf938c9b7a18558ccddb47b24a97541b3e712c32a92fd8492d597ed7e5ab: Status 404 returned error can't find the container with id 90afaf938c9b7a18558ccddb47b24a97541b3e712c32a92fd8492d597ed7e5ab
Jul 31 21:36:42.154331 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:42.154305    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-4wqtz"]
Jul 31 21:36:42.158593 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:42.158570    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a7ea88_8e73_44af_982d_adc960254ccc.slice/crio-3b7b074a12404efac214f78792c95a746c902ec19236cb75b61bce2a8277c22a WatchSource:0}: Error finding container 3b7b074a12404efac214f78792c95a746c902ec19236cb75b61bce2a8277c22a: Status 404 returned error can't find the container with id 3b7b074a12404efac214f78792c95a746c902ec19236cb75b61bce2a8277c22a
Jul 31 21:36:42.193821 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:42.193796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:42.193950 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:42.193933    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 21:36:42.194002 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:42.193993    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls podName:ce19efed-c49f-4620-abc9-6f312195bff0 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:43.193976466 +0000 UTC m=+43.456724807 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-8p8cq" (UID: "ce19efed-c49f-4620-abc9-6f312195bff0") : secret "cluster-monitoring-operator-tls" not found
Jul 31 21:36:42.395199 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:42.395097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:42.395199 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:42.395155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:42.395370 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:42.395245    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 21:36:42.395370 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:42.395260    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86b4d994cc-8wmt4: secret "image-registry-tls" not found
Jul 31 21:36:42.395370 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:42.395313    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls podName:8c8044c4-cc78-4d68-ac3a-888285c681d1 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:50.395298593 +0000 UTC m=+50.658046935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls") pod "image-registry-86b4d994cc-8wmt4" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1") : secret "image-registry-tls" not found
Jul 31 21:36:42.395370 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:42.395316    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 21:36:42.395370 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:42.395366    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls podName:7360b4d5-6dd2-467d-a3c8-75167012a942 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:50.395353345 +0000 UTC m=+50.658101687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls") pod "dns-default-xbrvr" (UID: "7360b4d5-6dd2-467d-a3c8-75167012a942") : secret "dns-default-metrics-tls" not found
Jul 31 21:36:42.514744 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:42.514705    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" event={"ID":"12a7ea88-8e73-44af-982d-adc960254ccc","Type":"ContainerStarted","Data":"3b7b074a12404efac214f78792c95a746c902ec19236cb75b61bce2a8277c22a"}
Jul 31 21:36:42.516103 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:42.516062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-dl4sm" event={"ID":"24c3514d-5367-43f1-aa6e-9afb94a11da7","Type":"ContainerStarted","Data":"803b4f711a2444bd523b97cb41d53fbe77fc5efc3a1957f274845621cef2c136"}
Jul 31 21:36:42.517227 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:42.517195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql" event={"ID":"26a15230-5d14-464e-bc26-9df69a8d5967","Type":"ContainerStarted","Data":"90afaf938c9b7a18558ccddb47b24a97541b3e712c32a92fd8492d597ed7e5ab"}
Jul 31 21:36:43.203019 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:43.202403    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:43.203019 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:43.202566    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 21:36:43.203019 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:43.202631    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls podName:ce19efed-c49f-4620-abc9-6f312195bff0 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:45.202612479 +0000 UTC m=+45.465360829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-8p8cq" (UID: "ce19efed-c49f-4620-abc9-6f312195bff0") : secret "cluster-monitoring-operator-tls" not found
Jul 31 21:36:45.219322 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:45.219283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:45.219786 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:45.219437    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 21:36:45.219786 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:45.219500    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls podName:ce19efed-c49f-4620-abc9-6f312195bff0 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:49.219483351 +0000 UTC m=+49.482231716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-8p8cq" (UID: "ce19efed-c49f-4620-abc9-6f312195bff0") : secret "cluster-monitoring-operator-tls" not found
Jul 31 21:36:46.529814 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:46.529363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql" event={"ID":"26a15230-5d14-464e-bc26-9df69a8d5967","Type":"ContainerStarted","Data":"09ca9600838ca2373a804933c65d16152839cb34db6af4e795b39389d68ed20c"}
Jul 31 21:36:46.531445 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:46.531232    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" event={"ID":"12a7ea88-8e73-44af-982d-adc960254ccc","Type":"ContainerStarted","Data":"cd6973f2d61f2a46897fc8ddcc54f87d8c17052ec0a4fdbaf36c80e56f99533e"}
Jul 31 21:36:46.531445 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:46.531289    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:46.532698 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:46.532665    2561 patch_prober.go:28] interesting pod/console-operator-5ccd6b445f-4wqtz container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body=
Jul 31 21:36:46.532795 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:46.532707    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" podUID="12a7ea88-8e73-44af-982d-adc960254ccc" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused"
Jul 31 21:36:46.545718 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:46.545666    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql" podStartSLOduration=1.277533839 podStartE2EDuration="5.545645708s" totalImagesPullingTime="4.268111869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:46.544735383 +0000 UTC m=+46.807483745" watchObservedRunningTime="2026-07-31 21:36:46.545645708 +0000 UTC m=+46.808394074"
Jul 31 21:36:46.560919 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:46.560863    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" podStartSLOduration=1.337143839 podStartE2EDuration="5.560841443s" totalImagesPullingTime="4.223697604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:46.560219621 +0000 UTC m=+46.822967983" watchObservedRunningTime="2026-07-31 21:36:46.560841443 +0000 UTC m=+46.823589807"
Jul 31 21:36:47.366279 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.366232    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4"]
Jul 31 21:36:47.411247 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.411214    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4"]
Jul 31 21:36:47.411397 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.411373    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4"
Jul 31 21:36:47.413751 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.413725    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 31 21:36:47.413901 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.413725    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mh49v\""
Jul 31 21:36:47.414113 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.414093    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 31 21:36:47.439392 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.439368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzkrf\" (UniqueName: \"kubernetes.io/projected/7cd3c871-741b-4e2c-949f-c2efa27f51b4-kube-api-access-kzkrf\") pod \"migrator-68bc745d87-v82d4\" (UID: \"7cd3c871-741b-4e2c-949f-c2efa27f51b4\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4"
Jul 31 21:36:47.539586 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.539558    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/0.log"
Jul 31 21:36:47.540028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.539606    2561 generic.go:356] "Generic (PLEG): container finished" podID="12a7ea88-8e73-44af-982d-adc960254ccc" containerID="cd6973f2d61f2a46897fc8ddcc54f87d8c17052ec0a4fdbaf36c80e56f99533e" exitCode=255
Jul 31 21:36:47.540028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.539690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" event={"ID":"12a7ea88-8e73-44af-982d-adc960254ccc","Type":"ContainerDied","Data":"cd6973f2d61f2a46897fc8ddcc54f87d8c17052ec0a4fdbaf36c80e56f99533e"}
Jul 31 21:36:47.540028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.539853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzkrf\" (UniqueName: \"kubernetes.io/projected/7cd3c871-741b-4e2c-949f-c2efa27f51b4-kube-api-access-kzkrf\") pod \"migrator-68bc745d87-v82d4\" (UID: \"7cd3c871-741b-4e2c-949f-c2efa27f51b4\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4"
Jul 31 21:36:47.540028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.539935    2561 scope.go:117] "RemoveContainer" containerID="cd6973f2d61f2a46897fc8ddcc54f87d8c17052ec0a4fdbaf36c80e56f99533e"
Jul 31 21:36:47.541613 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.541592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-dl4sm" event={"ID":"24c3514d-5367-43f1-aa6e-9afb94a11da7","Type":"ContainerStarted","Data":"274dab2ed0e208e6664cce832e895ea794520a3cbbfe0d45f4afa442ebba8d4a"}
Jul 31 21:36:47.575391 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.575371    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzkrf\" (UniqueName: \"kubernetes.io/projected/7cd3c871-741b-4e2c-949f-c2efa27f51b4-kube-api-access-kzkrf\") pod \"migrator-68bc745d87-v82d4\" (UID: \"7cd3c871-741b-4e2c-949f-c2efa27f51b4\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4"
Jul 31 21:36:47.651656 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.651601    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-dl4sm" podStartSLOduration=2.2660904 podStartE2EDuration="6.651580877s" totalImagesPullingTime="4.385490477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:47.649524646 +0000 UTC m=+47.912273022" watchObservedRunningTime="2026-07-31 21:36:47.651580877 +0000 UTC m=+47.914329242"
Jul 31 21:36:47.720200 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.720116    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4"
Jul 31 21:36:47.731734 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.731709    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5v7cq"]
Jul 31 21:36:47.745181 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.745162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.747492 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.747467    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 21:36:47.747605 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.747474    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 21:36:47.747605 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.747503    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 21:36:47.747802 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.747786    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 21:36:47.747847 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.747803    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zkn8l\""
Jul 31 21:36:47.753703 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.753650    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5v7cq"]
Jul 31 21:36:47.842143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.842101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-crio-socket\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.842288 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.842154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-data-volume\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.842334 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.842313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.842396 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.842377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.842450 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.842439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft88m\" (UniqueName: \"kubernetes.io/projected/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-api-access-ft88m\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.876645 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.876603    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4"]
Jul 31 21:36:47.880321 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:36:47.880288    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd3c871_741b_4e2c_949f_c2efa27f51b4.slice/crio-48716f18905818c13a902660096cd2053bab712999458df49909d3acb9030009 WatchSource:0}: Error finding container 48716f18905818c13a902660096cd2053bab712999458df49909d3acb9030009: Status 404 returned error can't find the container with id 48716f18905818c13a902660096cd2053bab712999458df49909d3acb9030009
Jul 31 21:36:47.943422 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.943338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.943422 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.943389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.943606 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:47.943468    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 21:36:47.943606 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.943529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft88m\" (UniqueName: \"kubernetes.io/projected/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-api-access-ft88m\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.943606 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:47.943550    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm podName:dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d nodeName:}" failed. No retries permitted until 2026-07-31 21:36:48.443531742 +0000 UTC m=+48.706280107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5v7cq" (UID: "dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d") : configmap "kube-rbac-proxy" not found
Jul 31 21:36:47.943606 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:47.943482    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 21:36:47.943606 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:47.943606    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls podName:dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d nodeName:}" failed. No retries permitted until 2026-07-31 21:36:48.443587921 +0000 UTC m=+48.706336263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v7cq" (UID: "dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d") : secret "insights-runtime-extractor-tls" not found
Jul 31 21:36:47.943776 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.943666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-crio-socket\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.943776 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.943699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-data-volume\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.943902 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.943886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-crio-socket\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.944017 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.944003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-data-volume\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:47.951720 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:47.951701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft88m\" (UniqueName: \"kubernetes.io/projected/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-api-access-ft88m\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:48.448027 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:48.447990    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:48.448209 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:48.448115    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 21:36:48.448209 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:48.448174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:48.448209 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:48.448203    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm podName:dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d nodeName:}" failed. No retries permitted until 2026-07-31 21:36:49.448182622 +0000 UTC m=+49.710930977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5v7cq" (UID: "dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d") : configmap "kube-rbac-proxy" not found
Jul 31 21:36:48.448406 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:48.448281    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 21:36:48.448406 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:48.448333    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls podName:dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d nodeName:}" failed. No retries permitted until 2026-07-31 21:36:49.44831799 +0000 UTC m=+49.711066350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v7cq" (UID: "dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d") : secret "insights-runtime-extractor-tls" not found
Jul 31 21:36:48.546129 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:48.546089    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4" event={"ID":"7cd3c871-741b-4e2c-949f-c2efa27f51b4","Type":"ContainerStarted","Data":"48716f18905818c13a902660096cd2053bab712999458df49909d3acb9030009"}
Jul 31 21:36:48.547845 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:48.547819    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/1.log"
Jul 31 21:36:48.548317 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:48.548294    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/0.log"
Jul 31 21:36:48.548456 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:48.548331    2561 generic.go:356] "Generic (PLEG): container finished" podID="12a7ea88-8e73-44af-982d-adc960254ccc" containerID="0072fd20e625c7c777a0ac979bdd8c25651466131e0feb578412a0613b4acf27" exitCode=255
Jul 31 21:36:48.548456 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:48.548436    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" event={"ID":"12a7ea88-8e73-44af-982d-adc960254ccc","Type":"ContainerDied","Data":"0072fd20e625c7c777a0ac979bdd8c25651466131e0feb578412a0613b4acf27"}
Jul 31 21:36:48.548569 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:48.548484    2561 scope.go:117] "RemoveContainer" containerID="cd6973f2d61f2a46897fc8ddcc54f87d8c17052ec0a4fdbaf36c80e56f99533e"
Jul 31 21:36:48.548706 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:48.548686    2561 scope.go:117] "RemoveContainer" containerID="0072fd20e625c7c777a0ac979bdd8c25651466131e0feb578412a0613b4acf27"
Jul 31 21:36:48.548897 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:48.548872    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4wqtz_openshift-console-operator(12a7ea88-8e73-44af-982d-adc960254ccc)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" podUID="12a7ea88-8e73-44af-982d-adc960254ccc"
Jul 31 21:36:49.254519 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:49.254478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:49.254704 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:49.254647    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 21:36:49.254750 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:49.254720    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls podName:ce19efed-c49f-4620-abc9-6f312195bff0 nodeName:}" failed. No retries permitted until 2026-07-31 21:36:57.254703761 +0000 UTC m=+57.517452103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-8p8cq" (UID: "ce19efed-c49f-4620-abc9-6f312195bff0") : secret "cluster-monitoring-operator-tls" not found
Jul 31 21:36:49.455756 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:49.455730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:49.455855 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:49.455832    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:49.455909 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:49.455886    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 21:36:49.455909 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:49.455896    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 21:36:49.455981 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:49.455941    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm podName:dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d nodeName:}" failed. No retries permitted until 2026-07-31 21:36:51.45592808 +0000 UTC m=+51.718676422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5v7cq" (UID: "dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d") : configmap "kube-rbac-proxy" not found
Jul 31 21:36:49.455981 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:49.455954    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls podName:dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d nodeName:}" failed. No retries permitted until 2026-07-31 21:36:51.45594813 +0000 UTC m=+51.718696472 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v7cq" (UID: "dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d") : secret "insights-runtime-extractor-tls" not found
Jul 31 21:36:49.551917 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:49.551888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4" event={"ID":"7cd3c871-741b-4e2c-949f-c2efa27f51b4","Type":"ContainerStarted","Data":"baa9dc67f14ec7b902b83e8749dc6b5f5af8b6df554d4bdbaaba539609e85377"}
Jul 31 21:36:49.553307 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:49.553287    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/1.log"
Jul 31 21:36:49.553637 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:49.553613    2561 scope.go:117] "RemoveContainer" containerID="0072fd20e625c7c777a0ac979bdd8c25651466131e0feb578412a0613b4acf27"
Jul 31 21:36:49.553768 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:49.553751    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4wqtz_openshift-console-operator(12a7ea88-8e73-44af-982d-adc960254ccc)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" podUID="12a7ea88-8e73-44af-982d-adc960254ccc"
Jul 31 21:36:49.863749 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:49.863678    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d2sf8_cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d/dns-node-resolver/0.log"
Jul 31 21:36:50.464854 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:50.464809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:36:50.465011 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:50.464918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:36:50.465011 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:50.464963    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 21:36:50.465011 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:50.464997    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 21:36:50.465011 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:50.465007    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86b4d994cc-8wmt4: secret "image-registry-tls" not found
Jul 31 21:36:50.465164 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:50.465024    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls podName:7360b4d5-6dd2-467d-a3c8-75167012a942 nodeName:}" failed. No retries permitted until 2026-07-31 21:37:06.465010096 +0000 UTC m=+66.727758439 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls") pod "dns-default-xbrvr" (UID: "7360b4d5-6dd2-467d-a3c8-75167012a942") : secret "dns-default-metrics-tls" not found
Jul 31 21:36:50.465164 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:50.465043    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls podName:8c8044c4-cc78-4d68-ac3a-888285c681d1 nodeName:}" failed. No retries permitted until 2026-07-31 21:37:06.465031789 +0000 UTC m=+66.727780132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls") pod "image-registry-86b4d994cc-8wmt4" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1") : secret "image-registry-tls" not found
Jul 31 21:36:50.558709 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:50.558673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4" event={"ID":"7cd3c871-741b-4e2c-949f-c2efa27f51b4","Type":"ContainerStarted","Data":"961f23aa0f3fc780be77e05b7dc45a77ed3de09dad1f5d351736a0e5e72a99eb"}
Jul 31 21:36:50.576700 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:50.576652    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-v82d4" podStartSLOduration=2.021020409 podStartE2EDuration="3.576636172s" totalImagesPullingTime="1.555615763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:36:50.575318466 +0000 UTC m=+50.838066827" watchObservedRunningTime="2026-07-31 21:36:50.576636172 +0000 UTC m=+50.839384536"
Jul 31 21:36:51.260757 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:51.260728    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sgfk2_00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e/node-ca/0.log"
Jul 31 21:36:51.474697 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:51.474661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:51.474876 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:51.474709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:51.474876 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:51.474798    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 21:36:51.474963 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:51.474927    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls podName:dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d nodeName:}" failed. No retries permitted until 2026-07-31 21:36:55.474904945 +0000 UTC m=+55.737653293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v7cq" (UID: "dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d") : secret "insights-runtime-extractor-tls" not found
Jul 31 21:36:51.475295 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:51.475278    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:51.860445 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:51.860413    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r9jbm_7ce0cdef-4292-4fa8-b7ff-c7fba201bbae/serve-healthcheck-canary/0.log"
Jul 31 21:36:51.982011 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:51.981977    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:51.982374 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:51.982361    2561 scope.go:117] "RemoveContainer" containerID="0072fd20e625c7c777a0ac979bdd8c25651466131e0feb578412a0613b4acf27"
Jul 31 21:36:51.982512 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:51.982493    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4wqtz_openshift-console-operator(12a7ea88-8e73-44af-982d-adc960254ccc)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" podUID="12a7ea88-8e73-44af-982d-adc960254ccc"
Jul 31 21:36:52.061054 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:52.061028    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-v82d4_7cd3c871-741b-4e2c-949f-c2efa27f51b4/migrator/0.log"
Jul 31 21:36:52.261046 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:52.260974    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-v82d4_7cd3c871-741b-4e2c-949f-c2efa27f51b4/graceful-termination/0.log"
Jul 31 21:36:52.460680 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:52.460652    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-v7rql_26a15230-5d14-464e-bc26-9df69a8d5967/kube-storage-version-migrator-operator/0.log"
Jul 31 21:36:55.505164 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:55.505115    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:36:55.505558 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:55.505267    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 21:36:55.505558 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:55.505328    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls podName:dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d nodeName:}" failed. No retries permitted until 2026-07-31 21:37:03.505312472 +0000 UTC m=+63.768060814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v7cq" (UID: "dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d") : secret "insights-runtime-extractor-tls" not found
Jul 31 21:36:56.532015 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:56.531982    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:36:56.532423 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:56.532362    2561 scope.go:117] "RemoveContainer" containerID="0072fd20e625c7c777a0ac979bdd8c25651466131e0feb578412a0613b4acf27"
Jul 31 21:36:56.532548 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:56.532527    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4wqtz_openshift-console-operator(12a7ea88-8e73-44af-982d-adc960254ccc)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" podUID="12a7ea88-8e73-44af-982d-adc960254ccc"
Jul 31 21:36:57.315694 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:36:57.315657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:36:57.315859 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:57.315806    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 21:36:57.315900 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:36:57.315871    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls podName:ce19efed-c49f-4620-abc9-6f312195bff0 nodeName:}" failed. No retries permitted until 2026-07-31 21:37:13.315855453 +0000 UTC m=+73.578603795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-8p8cq" (UID: "ce19efed-c49f-4620-abc9-6f312195bff0") : secret "cluster-monitoring-operator-tls" not found
Jul 31 21:37:00.489582 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:00.489555    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qfwgj"
Jul 31 21:37:03.567502 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:03.567465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:37:03.569713 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:03.569691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v7cq\" (UID: \"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d\") " pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:37:03.677042 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:03.677010    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zkn8l\""
Jul 31 21:37:03.685669 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:03.685650    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5v7cq"
Jul 31 21:37:03.808533 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:03.808501    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5v7cq"]
Jul 31 21:37:03.811359 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:03.811333    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6066a1_12cb_4ccd_b766_bf6c54ed1a0d.slice/crio-20c9c988b56f1fe6d2ff30e3ce9c916e5e32f1cfeec2e2e4a17071f6922b9c0e WatchSource:0}: Error finding container 20c9c988b56f1fe6d2ff30e3ce9c916e5e32f1cfeec2e2e4a17071f6922b9c0e: Status 404 returned error can't find the container with id 20c9c988b56f1fe6d2ff30e3ce9c916e5e32f1cfeec2e2e4a17071f6922b9c0e
Jul 31 21:37:04.600945 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:04.600918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v7cq" event={"ID":"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d","Type":"ContainerStarted","Data":"e8c855e5bdb4b8b09d8c4afad7ec4d5530bf4a344b901b4c95edd25b682bdc14"}
Jul 31 21:37:04.601226 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:04.600974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v7cq" event={"ID":"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d","Type":"ContainerStarted","Data":"b09662a515c0098aaeb9d1635b94c810c0caa1b275ab75d9c5c6fb8a76a9f2b0"}
Jul 31 21:37:04.601226 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:04.600992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v7cq" event={"ID":"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d","Type":"ContainerStarted","Data":"20c9c988b56f1fe6d2ff30e3ce9c916e5e32f1cfeec2e2e4a17071f6922b9c0e"}
Jul 31 21:37:06.090844 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.090758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tvh\" (UniqueName: \"kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh\") pod \"network-check-target-lwcrg\" (UID: \"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e\") " pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:37:06.090844 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.090795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:37:06.090844 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.090814    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:37:06.093065 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.093035    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 21:37:06.093436 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.093420    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 21:37:06.093545 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.093465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2tvh\" (UniqueName: \"kubernetes.io/projected/f626a3ca-8916-4bc6-8c74-71fa3b2cee8e-kube-api-access-x2tvh\") pod \"network-check-target-lwcrg\" (UID: \"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e\") " pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:37:06.103623 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.103601    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d760f73-be89-4905-88ed-68554584ed68-original-pull-secret\") pod \"global-pull-secret-syncer-rw7xc\" (UID: \"2d760f73-be89-4905-88ed-68554584ed68\") " pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:37:06.103721 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.103684    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d70e996-a47e-4761-b0c0-0795bc2bd2be-metrics-certs\") pod \"network-metrics-daemon-fpcs6\" (UID: \"6d70e996-a47e-4761-b0c0-0795bc2bd2be\") " pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:37:06.136119 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.136096    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw7xc"
Jul 31 21:37:06.144726 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.144705    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zxl4n\""
Jul 31 21:37:06.149978 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.149961    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2r85f\""
Jul 31 21:37:06.153459 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.153445    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpcs6"
Jul 31 21:37:06.158777 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.158750    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:37:06.286371 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.284200    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rw7xc"]
Jul 31 21:37:06.289968 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:06.289938    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d760f73_be89_4905_88ed_68554584ed68.slice/crio-3dd4612d116822f35b153b90c29bfcfaf6f92287ebd30cc63c28a20688f650bd WatchSource:0}: Error finding container 3dd4612d116822f35b153b90c29bfcfaf6f92287ebd30cc63c28a20688f650bd: Status 404 returned error can't find the container with id 3dd4612d116822f35b153b90c29bfcfaf6f92287ebd30cc63c28a20688f650bd
Jul 31 21:37:06.304811 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.304785    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fpcs6"]
Jul 31 21:37:06.307640 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:06.307617    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d70e996_a47e_4761_b0c0_0795bc2bd2be.slice/crio-d2765491b20ff14a670373ae011a1cc8854421437776d43bcb92b4bf49e7a91d WatchSource:0}: Error finding container d2765491b20ff14a670373ae011a1cc8854421437776d43bcb92b4bf49e7a91d: Status 404 returned error can't find the container with id d2765491b20ff14a670373ae011a1cc8854421437776d43bcb92b4bf49e7a91d
Jul 31 21:37:06.319952 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.319928    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lwcrg"]
Jul 31 21:37:06.323067 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:06.323047    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf626a3ca_8916_4bc6_8c74_71fa3b2cee8e.slice/crio-553c2d1b111c6432ee22a37a841afcc745b2babe099bf6ba69a66e1ec32188d3 WatchSource:0}: Error finding container 553c2d1b111c6432ee22a37a841afcc745b2babe099bf6ba69a66e1ec32188d3: Status 404 returned error can't find the container with id 553c2d1b111c6432ee22a37a841afcc745b2babe099bf6ba69a66e1ec32188d3
Jul 31 21:37:06.492632 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.492596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:37:06.492791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.492660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:37:06.494858 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.494831    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7360b4d5-6dd2-467d-a3c8-75167012a942-metrics-tls\") pod \"dns-default-xbrvr\" (UID: \"7360b4d5-6dd2-467d-a3c8-75167012a942\") " pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:37:06.495059 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.495040    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls\") pod \"image-registry-86b4d994cc-8wmt4\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") " pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:37:06.609579 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.609497    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v7cq" event={"ID":"dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d","Type":"ContainerStarted","Data":"e33e252e7f48f54d219f204efd10918129f42985e96b23ea6d3fce39420787e8"}
Jul 31 21:37:06.610661 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.610639    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fpcs6" event={"ID":"6d70e996-a47e-4761-b0c0-0795bc2bd2be","Type":"ContainerStarted","Data":"d2765491b20ff14a670373ae011a1cc8854421437776d43bcb92b4bf49e7a91d"}
Jul 31 21:37:06.611885 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.611865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lwcrg" event={"ID":"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e","Type":"ContainerStarted","Data":"72dd0287645d177d774f714bc3de151732e4c19f3cb486aa32680872eb2c2b51"}
Jul 31 21:37:06.611954 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.611906    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lwcrg" event={"ID":"f626a3ca-8916-4bc6-8c74-71fa3b2cee8e","Type":"ContainerStarted","Data":"553c2d1b111c6432ee22a37a841afcc745b2babe099bf6ba69a66e1ec32188d3"}
Jul 31 21:37:06.612011 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.611953    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:37:06.612855 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.612836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rw7xc" event={"ID":"2d760f73-be89-4905-88ed-68554584ed68","Type":"ContainerStarted","Data":"3dd4612d116822f35b153b90c29bfcfaf6f92287ebd30cc63c28a20688f650bd"}
Jul 31 21:37:06.627587 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.627550    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5v7cq" podStartSLOduration=17.745631589 podStartE2EDuration="19.62753881s" totalImagesPullingTime="1.881907221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:06.62695165 +0000 UTC m=+66.889700014" watchObservedRunningTime="2026-07-31 21:37:06.62753881 +0000 UTC m=+66.890287174"
Jul 31 21:37:06.642780 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.642679    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lwcrg" podStartSLOduration=66.642662615 podStartE2EDuration="1m6.642662615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:06.642250557 +0000 UTC m=+66.904998920" watchObservedRunningTime="2026-07-31 21:37:06.642662615 +0000 UTC m=+66.905410973"
Jul 31 21:37:06.711567 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.711537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dbtts\""
Jul 31 21:37:06.720022 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.720000    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:37:06.732071 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.732048    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jgwbq\""
Jul 31 21:37:06.741024 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.740994    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:37:06.877752 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.877643    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86b4d994cc-8wmt4"]
Jul 31 21:37:06.881458 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:06.881425    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8044c4_cc78_4d68_ac3a_888285c681d1.slice/crio-0ca366e6f452afc63b439865479ef3b27689f2ee69833e8a8e2df0b304a16240 WatchSource:0}: Error finding container 0ca366e6f452afc63b439865479ef3b27689f2ee69833e8a8e2df0b304a16240: Status 404 returned error can't find the container with id 0ca366e6f452afc63b439865479ef3b27689f2ee69833e8a8e2df0b304a16240
Jul 31 21:37:06.900780 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:06.900753    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xbrvr"]
Jul 31 21:37:06.907926 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:06.907901    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7360b4d5_6dd2_467d_a3c8_75167012a942.slice/crio-ef50fbdbe2521febb6af3a5b1cb778288c8c61bd50ce09e8f1daa288c7f958bb WatchSource:0}: Error finding container ef50fbdbe2521febb6af3a5b1cb778288c8c61bd50ce09e8f1daa288c7f958bb: Status 404 returned error can't find the container with id ef50fbdbe2521febb6af3a5b1cb778288c8c61bd50ce09e8f1daa288c7f958bb
Jul 31 21:37:07.619024 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:07.618945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbrvr" event={"ID":"7360b4d5-6dd2-467d-a3c8-75167012a942","Type":"ContainerStarted","Data":"ef50fbdbe2521febb6af3a5b1cb778288c8c61bd50ce09e8f1daa288c7f958bb"}
Jul 31 21:37:07.621276 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:07.620974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4" event={"ID":"8c8044c4-cc78-4d68-ac3a-888285c681d1","Type":"ContainerStarted","Data":"419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159"}
Jul 31 21:37:07.621276 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:07.621036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4" event={"ID":"8c8044c4-cc78-4d68-ac3a-888285c681d1","Type":"ContainerStarted","Data":"0ca366e6f452afc63b439865479ef3b27689f2ee69833e8a8e2df0b304a16240"}
Jul 31 21:37:07.621276 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:07.621272    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:37:07.639111 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:07.639034    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4" podStartSLOduration=66.639017787 podStartE2EDuration="1m6.639017787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:07.638522155 +0000 UTC m=+67.901270556" watchObservedRunningTime="2026-07-31 21:37:07.639017787 +0000 UTC m=+67.901766139"
Jul 31 21:37:08.627896 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:08.627855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fpcs6" event={"ID":"6d70e996-a47e-4761-b0c0-0795bc2bd2be","Type":"ContainerStarted","Data":"b02682ef1c7a4601b753411f3b78d23c2f092bd550b7d86fdfa491ed1b91214c"}
Jul 31 21:37:08.628355 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:08.627931    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fpcs6" event={"ID":"6d70e996-a47e-4761-b0c0-0795bc2bd2be","Type":"ContainerStarted","Data":"98a0f036d20933d379fa4b28e9e02277177d3db0ab917f21b4b48c78c6c45e3a"}
Jul 31 21:37:10.396035 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.395966    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fpcs6" podStartSLOduration=68.021956611 podStartE2EDuration="1m9.395944562s" totalImagesPullingTime="1.373987951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:08.654964629 +0000 UTC m=+68.917713003" watchObservedRunningTime="2026-07-31 21:37:10.395944562 +0000 UTC m=+70.658692927"
Jul 31 21:37:10.396883 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.396859    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582"]
Jul 31 21:37:10.399824 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.399802    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582"
Jul 31 21:37:10.401839 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.401813    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 31 21:37:10.403594 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.403572    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9824g\""
Jul 31 21:37:10.403694 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.403592    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 31 21:37:10.403694 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.403573    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 31 21:37:10.403694 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.403627    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 31 21:37:10.412923 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.412904    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582"]
Jul 31 21:37:10.531489 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.531457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/989964fc-8501-48c0-bc1f-471991f47d63-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-689d46595b-7z582\" (UID: \"989964fc-8501-48c0-bc1f-471991f47d63\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582"
Jul 31 21:37:10.531489 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.531500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wb2w\" (UniqueName: \"kubernetes.io/projected/989964fc-8501-48c0-bc1f-471991f47d63-kube-api-access-6wb2w\") pod \"managed-serviceaccount-addon-agent-689d46595b-7z582\" (UID: \"989964fc-8501-48c0-bc1f-471991f47d63\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582"
Jul 31 21:37:10.632539 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.632456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/989964fc-8501-48c0-bc1f-471991f47d63-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-689d46595b-7z582\" (UID: \"989964fc-8501-48c0-bc1f-471991f47d63\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582"
Jul 31 21:37:10.632539 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.632511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wb2w\" (UniqueName: \"kubernetes.io/projected/989964fc-8501-48c0-bc1f-471991f47d63-kube-api-access-6wb2w\") pod \"managed-serviceaccount-addon-agent-689d46595b-7z582\" (UID: \"989964fc-8501-48c0-bc1f-471991f47d63\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582"
Jul 31 21:37:10.635698 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.635641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/989964fc-8501-48c0-bc1f-471991f47d63-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-689d46595b-7z582\" (UID: \"989964fc-8501-48c0-bc1f-471991f47d63\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582"
Jul 31 21:37:10.650334 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.650266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wb2w\" (UniqueName: \"kubernetes.io/projected/989964fc-8501-48c0-bc1f-471991f47d63-kube-api-access-6wb2w\") pod \"managed-serviceaccount-addon-agent-689d46595b-7z582\" (UID: \"989964fc-8501-48c0-bc1f-471991f47d63\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582"
Jul 31 21:37:10.719018 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.718994    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582"
Jul 31 21:37:10.908570 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:10.908537    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod989964fc_8501_48c0_bc1f_471991f47d63.slice/crio-7276953bf1df98f3f43fd2421051de53dd8b3b162c3366b59c9af4b804f86c52 WatchSource:0}: Error finding container 7276953bf1df98f3f43fd2421051de53dd8b3b162c3366b59c9af4b804f86c52: Status 404 returned error can't find the container with id 7276953bf1df98f3f43fd2421051de53dd8b3b162c3366b59c9af4b804f86c52
Jul 31 21:37:10.915800 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:10.915777    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582"]
Jul 31 21:37:11.221773 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.221693    2561 scope.go:117] "RemoveContainer" containerID="0072fd20e625c7c777a0ac979bdd8c25651466131e0feb578412a0613b4acf27"
Jul 31 21:37:11.644512 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.644475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rw7xc" event={"ID":"2d760f73-be89-4905-88ed-68554584ed68","Type":"ContainerStarted","Data":"4bba82564fc4b2247963e7fe687f7bcf3d1e446dcbff49fe2837c243aff03520"}
Jul 31 21:37:11.646022 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.645999    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 21:37:11.646433 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.646416    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/1.log"
Jul 31 21:37:11.646485 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.646449    2561 generic.go:356] "Generic (PLEG): container finished" podID="12a7ea88-8e73-44af-982d-adc960254ccc" containerID="2cfafcfd8aa7310e38253dc8b74f669bde23bb2ff946dcab81fb87d00f9c1a62" exitCode=255
Jul 31 21:37:11.646538 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.646520    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" event={"ID":"12a7ea88-8e73-44af-982d-adc960254ccc","Type":"ContainerDied","Data":"2cfafcfd8aa7310e38253dc8b74f669bde23bb2ff946dcab81fb87d00f9c1a62"}
Jul 31 21:37:11.646591 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.646577    2561 scope.go:117] "RemoveContainer" containerID="0072fd20e625c7c777a0ac979bdd8c25651466131e0feb578412a0613b4acf27"
Jul 31 21:37:11.646840 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.646825    2561 scope.go:117] "RemoveContainer" containerID="2cfafcfd8aa7310e38253dc8b74f669bde23bb2ff946dcab81fb87d00f9c1a62"
Jul 31 21:37:11.647000 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:37:11.646981    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4wqtz_openshift-console-operator(12a7ea88-8e73-44af-982d-adc960254ccc)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" podUID="12a7ea88-8e73-44af-982d-adc960254ccc"
Jul 31 21:37:11.648596 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.648573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbrvr" event={"ID":"7360b4d5-6dd2-467d-a3c8-75167012a942","Type":"ContainerStarted","Data":"073f3adad79f0d0aa90260d4b0e0eb4e6370635f2343ed20326507a5bd3ae916"}
Jul 31 21:37:11.648695 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.648624    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:37:11.648695 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.648649    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbrvr" event={"ID":"7360b4d5-6dd2-467d-a3c8-75167012a942","Type":"ContainerStarted","Data":"cc0689fffca44d4cb26c55fbcc2cb4625d704b19f5a827fb80a3fa2ec42027fc"}
Jul 31 21:37:11.649560 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.649539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582" event={"ID":"989964fc-8501-48c0-bc1f-471991f47d63","Type":"ContainerStarted","Data":"7276953bf1df98f3f43fd2421051de53dd8b3b162c3366b59c9af4b804f86c52"}
Jul 31 21:37:11.658387 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.658350    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rw7xc" podStartSLOduration=66.230280474 podStartE2EDuration="1m10.658337649s" totalImagesPullingTime="4.428057175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:11.657532151 +0000 UTC m=+71.920280515" watchObservedRunningTime="2026-07-31 21:37:11.658337649 +0000 UTC m=+71.921086013"
Jul 31 21:37:11.673706 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.673665    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xbrvr" podStartSLOduration=33.912765219 podStartE2EDuration="37.673654167s" totalImagesPullingTime="3.760888948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:11.673350489 +0000 UTC m=+71.936098853" watchObservedRunningTime="2026-07-31 21:37:11.673654167 +0000 UTC m=+71.936402531"
Jul 31 21:37:11.982897 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:11.982806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:37:12.655116 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:12.655085    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 21:37:12.655568 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:12.655514    2561 scope.go:117] "RemoveContainer" containerID="2cfafcfd8aa7310e38253dc8b74f669bde23bb2ff946dcab81fb87d00f9c1a62"
Jul 31 21:37:12.655717 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:37:12.655691    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4wqtz_openshift-console-operator(12a7ea88-8e73-44af-982d-adc960254ccc)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" podUID="12a7ea88-8e73-44af-982d-adc960254ccc"
Jul 31 21:37:13.355460 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:13.355420    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:37:13.358148 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:13.358104    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce19efed-c49f-4620-abc9-6f312195bff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-8p8cq\" (UID: \"ce19efed-c49f-4620-abc9-6f312195bff0\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:37:13.381386 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:13.381357    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rc6mk\""
Jul 31 21:37:13.389712 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:13.389696    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"
Jul 31 21:37:13.862831 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:13.862802    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq"]
Jul 31 21:37:13.866681 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:13.866655    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce19efed_c49f_4620_abc9_6f312195bff0.slice/crio-644316d9d58fd7349147f3ec3f574cfdd59725168031ab949369e862b135ce8f WatchSource:0}: Error finding container 644316d9d58fd7349147f3ec3f574cfdd59725168031ab949369e862b135ce8f: Status 404 returned error can't find the container with id 644316d9d58fd7349147f3ec3f574cfdd59725168031ab949369e862b135ce8f
Jul 31 21:37:14.664475 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:14.664434    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582" event={"ID":"989964fc-8501-48c0-bc1f-471991f47d63","Type":"ContainerStarted","Data":"ac11da63c32ebc4188383eeef962560c4033ac54521b2f2629b83f6ead8ec0ce"}
Jul 31 21:37:14.665756 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:14.665730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq" event={"ID":"ce19efed-c49f-4620-abc9-6f312195bff0","Type":"ContainerStarted","Data":"644316d9d58fd7349147f3ec3f574cfdd59725168031ab949369e862b135ce8f"}
Jul 31 21:37:14.684763 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:14.684697    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-689d46595b-7z582" podStartSLOduration=1.795797844 podStartE2EDuration="4.684683253s" totalImagesPullingTime="2.888885409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:14.683910133 +0000 UTC m=+74.946658499" watchObservedRunningTime="2026-07-31 21:37:14.684683253 +0000 UTC m=+74.947431616"
Jul 31 21:37:16.532391 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:16.532352    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:37:16.532776 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:16.532763    2561 scope.go:117] "RemoveContainer" containerID="2cfafcfd8aa7310e38253dc8b74f669bde23bb2ff946dcab81fb87d00f9c1a62"
Jul 31 21:37:16.532929 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:37:16.532911    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4wqtz_openshift-console-operator(12a7ea88-8e73-44af-982d-adc960254ccc)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" podUID="12a7ea88-8e73-44af-982d-adc960254ccc"
Jul 31 21:37:16.673785 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:16.673752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq" event={"ID":"ce19efed-c49f-4620-abc9-6f312195bff0","Type":"ContainerStarted","Data":"7200675e5810b83a4a6104702da48ea99f696a5ba88742d88acc210532c1fab5"}
Jul 31 21:37:16.688938 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:16.688888    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-8p8cq" podStartSLOduration=33.644512401 podStartE2EDuration="35.688871626s" totalImagesPullingTime="2.044359225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:16.688090871 +0000 UTC m=+76.950839235" watchObservedRunningTime="2026-07-31 21:37:16.688871626 +0000 UTC m=+76.951619989"
Jul 31 21:37:21.657650 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:21.657619    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xbrvr"
Jul 31 21:37:22.891255 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.891125    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"]
Jul 31 21:37:22.895480 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.895461    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:22.897686 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.897664    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 31 21:37:22.897686 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.897682    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 31 21:37:22.897856 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.897787    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 21:37:22.898119 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.898104    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9npd6\""
Jul 31 21:37:22.907275 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.907253    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"]
Jul 31 21:37:22.929697 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.929676    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2nhjr"]
Jul 31 21:37:22.933090 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.933069    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:22.935327 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.935311    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 21:37:22.935422 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.935338    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 21:37:22.935422 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.935341    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 21:37:22.935647 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:22.935630    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6wjct\""
Jul 31 21:37:23.024831 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.024803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6fw\" (UniqueName: \"kubernetes.io/projected/a92fc535-e796-42d3-a398-84343d498630-kube-api-access-7g6fw\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.024831 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.024835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1845b24a-5171-4694-9185-4cb29e3b1298-root\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.025048 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.024872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-wtmp\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.025048 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.024895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.025048 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.024987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92fc535-e796-42d3-a398-84343d498630-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.025048 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.025034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1845b24a-5171-4694-9185-4cb29e3b1298-sys\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.025264 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.025055    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.025264 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.025075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbkxj\" (UniqueName: \"kubernetes.io/projected/1845b24a-5171-4694-9185-4cb29e3b1298-kube-api-access-cbkxj\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.025264 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.025129    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1845b24a-5171-4694-9185-4cb29e3b1298-metrics-client-ca\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.025264 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.025215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a92fc535-e796-42d3-a398-84343d498630-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.025264 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.025233    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-tls\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.025264 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.025254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a92fc535-e796-42d3-a398-84343d498630-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.025463 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.025270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-textfile\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.126439 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a92fc535-e796-42d3-a398-84343d498630-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.126439 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-tls\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.126683 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a92fc535-e796-42d3-a398-84343d498630-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.126683 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-textfile\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.126683 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6fw\" (UniqueName: \"kubernetes.io/projected/a92fc535-e796-42d3-a398-84343d498630-kube-api-access-7g6fw\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.126683 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:37:23.126631    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 31 21:37:23.126893 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1845b24a-5171-4694-9185-4cb29e3b1298-root\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.126893 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:37:23.126693    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-tls podName:1845b24a-5171-4694-9185-4cb29e3b1298 nodeName:}" failed. No retries permitted until 2026-07-31 21:37:23.626673046 +0000 UTC m=+83.889421391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-tls") pod "node-exporter-2nhjr" (UID: "1845b24a-5171-4694-9185-4cb29e3b1298") : secret "node-exporter-tls" not found
Jul 31 21:37:23.126893 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1845b24a-5171-4694-9185-4cb29e3b1298-root\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.126893 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-wtmp\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.126893 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.126893 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126849    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92fc535-e796-42d3-a398-84343d498630-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.126893 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1845b24a-5171-4694-9185-4cb29e3b1298-sys\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.127272 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-textfile\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.127272 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.127272 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbkxj\" (UniqueName: \"kubernetes.io/projected/1845b24a-5171-4694-9185-4cb29e3b1298-kube-api-access-cbkxj\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.127272 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-wtmp\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.127272 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:37:23.126988    2561 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 31 21:37:23.127272 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.126993    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1845b24a-5171-4694-9185-4cb29e3b1298-metrics-client-ca\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.127272 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:37:23.127054    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92fc535-e796-42d3-a398-84343d498630-openshift-state-metrics-tls podName:a92fc535-e796-42d3-a398-84343d498630 nodeName:}" failed. No retries permitted until 2026-07-31 21:37:23.627034523 +0000 UTC m=+83.889782865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a92fc535-e796-42d3-a398-84343d498630-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-wtx6h" (UID: "a92fc535-e796-42d3-a398-84343d498630") : secret "openshift-state-metrics-tls" not found
Jul 31 21:37:23.127559 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.127332    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a92fc535-e796-42d3-a398-84343d498630-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.127559 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.127402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1845b24a-5171-4694-9185-4cb29e3b1298-sys\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.127559 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.127426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.127559 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.127537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1845b24a-5171-4694-9185-4cb29e3b1298-metrics-client-ca\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.129715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.129692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.129868 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.129847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a92fc535-e796-42d3-a398-84343d498630-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.134585 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.134560    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6fw\" (UniqueName: \"kubernetes.io/projected/a92fc535-e796-42d3-a398-84343d498630-kube-api-access-7g6fw\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.134693 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.134565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbkxj\" (UniqueName: \"kubernetes.io/projected/1845b24a-5171-4694-9185-4cb29e3b1298-kube-api-access-cbkxj\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.630823 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.630786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-tls\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.631018 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.630854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92fc535-e796-42d3-a398-84343d498630-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.633193 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.633160    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1845b24a-5171-4694-9185-4cb29e3b1298-node-exporter-tls\") pod \"node-exporter-2nhjr\" (UID: \"1845b24a-5171-4694-9185-4cb29e3b1298\") " pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.633304 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.633248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92fc535-e796-42d3-a398-84343d498630-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wtx6h\" (UID: \"a92fc535-e796-42d3-a398-84343d498630\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.804585 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.804546    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"
Jul 31 21:37:23.842301 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.842271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2nhjr"
Jul 31 21:37:23.854813 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:23.854766    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1845b24a_5171_4694_9185_4cb29e3b1298.slice/crio-fd91045ce996deead9f1ed11115f0eb9017f0db4647021c430739ea293ee70dd WatchSource:0}: Error finding container fd91045ce996deead9f1ed11115f0eb9017f0db4647021c430739ea293ee70dd: Status 404 returned error can't find the container with id fd91045ce996deead9f1ed11115f0eb9017f0db4647021c430739ea293ee70dd
Jul 31 21:37:23.939264 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:23.939232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h"]
Jul 31 21:37:23.943050 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:23.943024    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92fc535_e796_42d3_a398_84343d498630.slice/crio-a411b5e4796fdf69b0676347f697c3277eb4f40fae58036c26544f2ed2378b3d WatchSource:0}: Error finding container a411b5e4796fdf69b0676347f697c3277eb4f40fae58036c26544f2ed2378b3d: Status 404 returned error can't find the container with id a411b5e4796fdf69b0676347f697c3277eb4f40fae58036c26544f2ed2378b3d
Jul 31 21:37:24.704014 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:24.703981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h" event={"ID":"a92fc535-e796-42d3-a398-84343d498630","Type":"ContainerStarted","Data":"2c4d4e915720ecf010e3f7d76245f3b26cf8aaa159411971814fd7967c2808f0"}
Jul 31 21:37:24.704110 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:24.704051    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h" event={"ID":"a92fc535-e796-42d3-a398-84343d498630","Type":"ContainerStarted","Data":"14e356aeffa518e847f90184f222e14bfb1c1da171e541a5e0c7e08dbd0d9658"}
Jul 31 21:37:24.704110 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:24.704078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h" event={"ID":"a92fc535-e796-42d3-a398-84343d498630","Type":"ContainerStarted","Data":"a411b5e4796fdf69b0676347f697c3277eb4f40fae58036c26544f2ed2378b3d"}
Jul 31 21:37:24.705273 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:24.705214    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nhjr" event={"ID":"1845b24a-5171-4694-9185-4cb29e3b1298","Type":"ContainerStarted","Data":"fd91045ce996deead9f1ed11115f0eb9017f0db4647021c430739ea293ee70dd"}
Jul 31 21:37:25.710690 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:25.710656    2561 generic.go:356] "Generic (PLEG): container finished" podID="1845b24a-5171-4694-9185-4cb29e3b1298" containerID="7ca8adfe104ce1135087c2aefd2c0f82d5dd29efca6916f8e7d41d5d5df7df48" exitCode=0
Jul 31 21:37:25.711185 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:25.710743    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nhjr" event={"ID":"1845b24a-5171-4694-9185-4cb29e3b1298","Type":"ContainerDied","Data":"7ca8adfe104ce1135087c2aefd2c0f82d5dd29efca6916f8e7d41d5d5df7df48"}
Jul 31 21:37:25.713034 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:25.713011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h" event={"ID":"a92fc535-e796-42d3-a398-84343d498630","Type":"ContainerStarted","Data":"76c01f3fdf3fc9718e3fd83a97357748486abf36bfb63d81365ba4c81aa7dfc6"}
Jul 31 21:37:25.744190 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:25.744122    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wtx6h" podStartSLOduration=2.5829205379999998 podStartE2EDuration="3.744101168s" totalImagesPullingTime="1.16118063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:25.743372791 +0000 UTC m=+86.006121155" watchObservedRunningTime="2026-07-31 21:37:25.744101168 +0000 UTC m=+86.006849533"
Jul 31 21:37:26.718681 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:26.718640    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nhjr" event={"ID":"1845b24a-5171-4694-9185-4cb29e3b1298","Type":"ContainerStarted","Data":"90a6f8599c9b606175419224dc7d8163e43771c99903ababfc33bc8a657fba29"}
Jul 31 21:37:26.719081 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:26.718696    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nhjr" event={"ID":"1845b24a-5171-4694-9185-4cb29e3b1298","Type":"ContainerStarted","Data":"803361248aee376ee77af3e3ca97c79a0b37f5575ecdb091fb683b09f4e2199a"}
Jul 31 21:37:26.724989 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:26.724964    2561 patch_prober.go:28] interesting pod/image-registry-86b4d994cc-8wmt4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 31 21:37:26.725125 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:26.725007    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4" podUID="8c8044c4-cc78-4d68-ac3a-888285c681d1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 21:37:26.737880 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:26.737838    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2nhjr" podStartSLOduration=3.844910808 podStartE2EDuration="4.737822006s" totalImagesPullingTime="827.521045ms" totalInitContainerRuntime="65.390153ms" isStatefulPod=true podCreationTimestamp="2026-07-31 21:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:26.736965741 +0000 UTC m=+86.999714100" watchObservedRunningTime="2026-07-31 21:37:26.737822006 +0000 UTC m=+87.000570370"
Jul 31 21:37:27.197218 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.197183    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-86b995f864-xcxh9"]
Jul 31 21:37:27.200467 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.200448    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.203448 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.203420    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-q7fcd\""
Jul 31 21:37:27.203567 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.203449    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 31 21:37:27.203567 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.203465    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 31 21:37:27.203567 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.203449    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 31 21:37:27.203567 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.203420    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1kgtg8t8gakp4\""
Jul 31 21:37:27.203567 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.203456    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 31 21:37:27.213068 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.213040    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86b995f864-xcxh9"]
Jul 31 21:37:27.361904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.361872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/648ef71b-d43f-4064-8809-eab70495c995-metrics-server-audit-profiles\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.362068 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.361917    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/648ef71b-d43f-4064-8809-eab70495c995-secret-metrics-server-client-certs\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.362068 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.361942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/648ef71b-d43f-4064-8809-eab70495c995-secret-metrics-server-tls\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.362068 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.362043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/648ef71b-d43f-4064-8809-eab70495c995-audit-log\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.362216 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.362087    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5cx\" (UniqueName: \"kubernetes.io/projected/648ef71b-d43f-4064-8809-eab70495c995-kube-api-access-sg5cx\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.362216 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.362115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/648ef71b-d43f-4064-8809-eab70495c995-client-ca-bundle\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.362216 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.362149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648ef71b-d43f-4064-8809-eab70495c995-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.462895 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.462819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/648ef71b-d43f-4064-8809-eab70495c995-secret-metrics-server-client-certs\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.462895 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.462877    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/648ef71b-d43f-4064-8809-eab70495c995-secret-metrics-server-tls\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.463122 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.462919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/648ef71b-d43f-4064-8809-eab70495c995-audit-log\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.463122 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.462968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5cx\" (UniqueName: \"kubernetes.io/projected/648ef71b-d43f-4064-8809-eab70495c995-kube-api-access-sg5cx\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.463122 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.463011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/648ef71b-d43f-4064-8809-eab70495c995-client-ca-bundle\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.463122 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.463040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648ef71b-d43f-4064-8809-eab70495c995-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.463122 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.463071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/648ef71b-d43f-4064-8809-eab70495c995-metrics-server-audit-profiles\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.463459 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.463436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/648ef71b-d43f-4064-8809-eab70495c995-audit-log\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.463893 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.463864    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648ef71b-d43f-4064-8809-eab70495c995-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.464712 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.464682    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/648ef71b-d43f-4064-8809-eab70495c995-metrics-server-audit-profiles\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.465440 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.465418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/648ef71b-d43f-4064-8809-eab70495c995-secret-metrics-server-client-certs\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.465546 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.465479    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/648ef71b-d43f-4064-8809-eab70495c995-client-ca-bundle\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.465709 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.465689    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/648ef71b-d43f-4064-8809-eab70495c995-secret-metrics-server-tls\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.470851 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.470827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5cx\" (UniqueName: \"kubernetes.io/projected/648ef71b-d43f-4064-8809-eab70495c995-kube-api-access-sg5cx\") pod \"metrics-server-86b995f864-xcxh9\" (UID: \"648ef71b-d43f-4064-8809-eab70495c995\") " pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.509687 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.509661    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:27.644847 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.644816    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86b995f864-xcxh9"]
Jul 31 21:37:27.647759 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:27.647732    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648ef71b_d43f_4064_8809_eab70495c995.slice/crio-b32f885280770b63b35ab0eae120078886fb9d6299c55572d3dd0928ddc4e2bd WatchSource:0}: Error finding container b32f885280770b63b35ab0eae120078886fb9d6299c55572d3dd0928ddc4e2bd: Status 404 returned error can't find the container with id b32f885280770b63b35ab0eae120078886fb9d6299c55572d3dd0928ddc4e2bd
Jul 31 21:37:27.654673 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.654652    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"]
Jul 31 21:37:27.658771 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.658755    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"
Jul 31 21:37:27.660732 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.660714    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 31 21:37:27.661059 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.661046    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tk94h\""
Jul 31 21:37:27.667387 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.667362    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"]
Jul 31 21:37:27.723625 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.723543    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86b995f864-xcxh9" event={"ID":"648ef71b-d43f-4064-8809-eab70495c995","Type":"ContainerStarted","Data":"b32f885280770b63b35ab0eae120078886fb9d6299c55572d3dd0928ddc4e2bd"}
Jul 31 21:37:27.765272 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.765241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75b6m\" (UniqueName: \"kubernetes.io/projected/59e2e80a-f0c0-4a29-9238-13ecfff3e8c5-kube-api-access-75b6m\") pod \"monitoring-plugin-6dffbbd67-t6c86\" (UID: \"59e2e80a-f0c0-4a29-9238-13ecfff3e8c5\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"
Jul 31 21:37:27.765397 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.765304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59e2e80a-f0c0-4a29-9238-13ecfff3e8c5-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-t6c86\" (UID: \"59e2e80a-f0c0-4a29-9238-13ecfff3e8c5\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"
Jul 31 21:37:27.866009 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.865979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59e2e80a-f0c0-4a29-9238-13ecfff3e8c5-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-t6c86\" (UID: \"59e2e80a-f0c0-4a29-9238-13ecfff3e8c5\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"
Jul 31 21:37:27.866216 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.866039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75b6m\" (UniqueName: \"kubernetes.io/projected/59e2e80a-f0c0-4a29-9238-13ecfff3e8c5-kube-api-access-75b6m\") pod \"monitoring-plugin-6dffbbd67-t6c86\" (UID: \"59e2e80a-f0c0-4a29-9238-13ecfff3e8c5\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"
Jul 31 21:37:27.866216 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:37:27.866167    2561 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 31 21:37:27.866391 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:37:27.866234    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59e2e80a-f0c0-4a29-9238-13ecfff3e8c5-monitoring-plugin-cert podName:59e2e80a-f0c0-4a29-9238-13ecfff3e8c5 nodeName:}" failed. No retries permitted until 2026-07-31 21:37:28.36621222 +0000 UTC m=+88.628960583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/59e2e80a-f0c0-4a29-9238-13ecfff3e8c5-monitoring-plugin-cert") pod "monitoring-plugin-6dffbbd67-t6c86" (UID: "59e2e80a-f0c0-4a29-9238-13ecfff3e8c5") : secret "monitoring-plugin-cert" not found
Jul 31 21:37:27.875658 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:27.875636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75b6m\" (UniqueName: \"kubernetes.io/projected/59e2e80a-f0c0-4a29-9238-13ecfff3e8c5-kube-api-access-75b6m\") pod \"monitoring-plugin-6dffbbd67-t6c86\" (UID: \"59e2e80a-f0c0-4a29-9238-13ecfff3e8c5\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"
Jul 31 21:37:28.370800 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:28.370763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59e2e80a-f0c0-4a29-9238-13ecfff3e8c5-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-t6c86\" (UID: \"59e2e80a-f0c0-4a29-9238-13ecfff3e8c5\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"
Jul 31 21:37:28.373536 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:28.373512    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59e2e80a-f0c0-4a29-9238-13ecfff3e8c5-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-t6c86\" (UID: \"59e2e80a-f0c0-4a29-9238-13ecfff3e8c5\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"
Jul 31 21:37:28.568730 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:28.568692    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"
Jul 31 21:37:28.633198 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:28.633113    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:37:28.710045 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:28.710004    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"]
Jul 31 21:37:28.977093 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:28.977010    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e2e80a_f0c0_4a29_9238_13ecfff3e8c5.slice/crio-72e2cf3103d4ec1884415d014f2bab3c23912cc1af2bdf54052b229c5abec0a2 WatchSource:0}: Error finding container 72e2cf3103d4ec1884415d014f2bab3c23912cc1af2bdf54052b229c5abec0a2: Status 404 returned error can't find the container with id 72e2cf3103d4ec1884415d014f2bab3c23912cc1af2bdf54052b229c5abec0a2
Jul 31 21:37:29.193606 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.193557    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 21:37:29.197433 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.197410    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.200206 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200000    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 31 21:37:29.200206 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200022    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 31 21:37:29.200206 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200028    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 31 21:37:29.200206 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200041    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 31 21:37:29.200206 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200063    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 31 21:37:29.200206 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200034    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 31 21:37:29.200608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200317    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5b7qt98vjohdq\""
Jul 31 21:37:29.200608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200417    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 31 21:37:29.200608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200454    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 31 21:37:29.200608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200464    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 31 21:37:29.200608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200534    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 31 21:37:29.200608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200560    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 31 21:37:29.200608 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200599    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 31 21:37:29.200916 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200676    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 31 21:37:29.200916 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.200781    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-p95rf\""
Jul 31 21:37:29.204628 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.204608    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 31 21:37:29.218634 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.218606    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 21:37:29.222243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.222162    2561 scope.go:117] "RemoveContainer" containerID="2cfafcfd8aa7310e38253dc8b74f669bde23bb2ff946dcab81fb87d00f9c1a62"
Jul 31 21:37:29.222474 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:37:29.222404    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-4wqtz_openshift-console-operator(12a7ea88-8e73-44af-982d-adc960254ccc)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" podUID="12a7ea88-8e73-44af-982d-adc960254ccc"
Jul 31 21:37:29.378180 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378373 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378373 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378259    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378373 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378373 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378373 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378493    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kc8c\" (UniqueName: \"kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-kube-api-access-4kc8c\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378948 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378948 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378948 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378753    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378948 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378948 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378791    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378948 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.378948 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.378887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.479461 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479425    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.479461 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.479691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479492    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.479691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.479691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.479691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.479691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479581    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.479691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc8c\" (UniqueName: \"kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-kube-api-access-4kc8c\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.479691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.479691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479888    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480066 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.479996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480613 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.480515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.480918 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.480894    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.481074 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.481014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.481662 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.481637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.482597 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.482575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.482731 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.482703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.482905 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.482886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.483705 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.483401    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.483705 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.483415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.483705 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.483507    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.484580 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.484511    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.484743 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.484718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.484949 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.484920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.485987 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.485923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.486082 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.486059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.486221 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.486192    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.486398 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.486379    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.486504 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.486482    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.487935 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.487915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kc8c\" (UniqueName: \"kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-kube-api-access-4kc8c\") pod \"prometheus-k8s-0\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.508080 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.508053    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:29.667744 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.667613    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 21:37:29.671826 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:37:29.671791    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d825328_92c9_45ab_b16a_f4aa8b613a7d.slice/crio-937388336bf30370dbe189f9206840f951a76b7c25c84855b6c50c2af1c2e923 WatchSource:0}: Error finding container 937388336bf30370dbe189f9206840f951a76b7c25c84855b6c50c2af1c2e923: Status 404 returned error can't find the container with id 937388336bf30370dbe189f9206840f951a76b7c25c84855b6c50c2af1c2e923
Jul 31 21:37:29.735897 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.735812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86b995f864-xcxh9" event={"ID":"648ef71b-d43f-4064-8809-eab70495c995","Type":"ContainerStarted","Data":"5527cb4da9fa0e0b2bbeec0e421555ca39cca4aa21268b442b90c0b61923252c"}
Jul 31 21:37:29.737515 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.737479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerStarted","Data":"937388336bf30370dbe189f9206840f951a76b7c25c84855b6c50c2af1c2e923"}
Jul 31 21:37:29.738866 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.738836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86" event={"ID":"59e2e80a-f0c0-4a29-9238-13ecfff3e8c5","Type":"ContainerStarted","Data":"72e2cf3103d4ec1884415d014f2bab3c23912cc1af2bdf54052b229c5abec0a2"}
Jul 31 21:37:29.754446 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:29.754382    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-86b995f864-xcxh9" podStartSLOduration=1.3819547110000001 podStartE2EDuration="2.754368202s" totalImagesPullingTime="1.372413491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:29.754209093 +0000 UTC m=+90.016957451" watchObservedRunningTime="2026-07-31 21:37:29.754368202 +0000 UTC m=+90.017116565"
Jul 31 21:37:30.751312 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:30.751223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86" event={"ID":"59e2e80a-f0c0-4a29-9238-13ecfff3e8c5","Type":"ContainerStarted","Data":"51f897e9706a9a28c6032f1c203db08c84c92067690227abb4554385e316311e"}
Jul 31 21:37:30.766866 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:30.766817    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86" podStartSLOduration=2.343271709 podStartE2EDuration="3.766799867s" totalImagesPullingTime="1.423528158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:30.765334765 +0000 UTC m=+91.028083131" watchObservedRunningTime="2026-07-31 21:37:30.766799867 +0000 UTC m=+91.029548258"
Jul 31 21:37:31.756405 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:31.756371    2561 generic.go:356] "Generic (PLEG): container finished" podID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerID="866ae0a75d1c5ffb5e5d382b59c1f8f17a0696f5a443e2eae43a0061112b3d28" exitCode=0
Jul 31 21:37:31.756807 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:31.756450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerDied","Data":"866ae0a75d1c5ffb5e5d382b59c1f8f17a0696f5a443e2eae43a0061112b3d28"}
Jul 31 21:37:31.756807 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:31.756737    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"
Jul 31 21:37:31.761688 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:31.761666    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6c86"
Jul 31 21:37:33.132284 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:33.132249    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86b4d994cc-8wmt4"]
Jul 31 21:37:34.772440 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:34.772403    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerStarted","Data":"c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b"}
Jul 31 21:37:34.772738 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:34.772532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerStarted","Data":"b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda"}
Jul 31 21:37:36.783399 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:36.783358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerStarted","Data":"f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd"}
Jul 31 21:37:36.783766 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:36.783441    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerStarted","Data":"e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e"}
Jul 31 21:37:36.783766 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:36.783484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerStarted","Data":"1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a"}
Jul 31 21:37:36.783766 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:36.783528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerStarted","Data":"7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3"}
Jul 31 21:37:36.809639 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:36.809585    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.889769364 podStartE2EDuration="7.809568343s" totalImagesPullingTime="5.903849577s" totalInitContainerRuntime="15.949402ms" isStatefulPod=false podCreationTimestamp="2026-07-31 21:37:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:37:36.808744818 +0000 UTC m=+97.071493219" watchObservedRunningTime="2026-07-31 21:37:36.809568343 +0000 UTC m=+97.072316727"
Jul 31 21:37:37.624181 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:37.624129    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lwcrg"
Jul 31 21:37:39.509226 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:39.509188    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:37:44.221853 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:44.221811    2561 scope.go:117] "RemoveContainer" containerID="2cfafcfd8aa7310e38253dc8b74f669bde23bb2ff946dcab81fb87d00f9c1a62"
Jul 31 21:37:44.817401 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:44.817374    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 21:37:44.817579 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:44.817491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz" event={"ID":"12a7ea88-8e73-44af-982d-adc960254ccc","Type":"ContainerStarted","Data":"aff5fa9fe9a2adc8d770f6e4d751ba2eaa3dbaa99dd4ca66cc37c6533cfda511"}
Jul 31 21:37:44.817827 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:44.817796    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:37:44.823210 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:44.823184    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-4wqtz"
Jul 31 21:37:47.509929 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:47.509882    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:47.510318 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:47.509950    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:37:56.868542 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:56.868508    2561 generic.go:356] "Generic (PLEG): container finished" podID="26a15230-5d14-464e-bc26-9df69a8d5967" containerID="09ca9600838ca2373a804933c65d16152839cb34db6af4e795b39389d68ed20c" exitCode=0
Jul 31 21:37:56.868907 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:56.868582    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql" event={"ID":"26a15230-5d14-464e-bc26-9df69a8d5967","Type":"ContainerDied","Data":"09ca9600838ca2373a804933c65d16152839cb34db6af4e795b39389d68ed20c"}
Jul 31 21:37:56.868907 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:56.868903    2561 scope.go:117] "RemoveContainer" containerID="09ca9600838ca2373a804933c65d16152839cb34db6af4e795b39389d68ed20c"
Jul 31 21:37:57.873747 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:57.873712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v7rql" event={"ID":"26a15230-5d14-464e-bc26-9df69a8d5967","Type":"ContainerStarted","Data":"2105e87913a8af0e390604563b03eafd619f6b19a404e0ebfb5dc86d6a88d099"}
Jul 31 21:37:58.154755 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.154652    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4" podUID="8c8044c4-cc78-4d68-ac3a-888285c681d1" containerName="registry" containerID="cri-o://419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159" gracePeriod=30
Jul 31 21:37:58.389311 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.389287    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:37:58.535405 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.535378    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-trusted-ca\") pod \"8c8044c4-cc78-4d68-ac3a-888285c681d1\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") "
Jul 31 21:37:58.535559 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.535417    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-certificates\") pod \"8c8044c4-cc78-4d68-ac3a-888285c681d1\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") "
Jul 31 21:37:58.535559 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.535442    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-bound-sa-token\") pod \"8c8044c4-cc78-4d68-ac3a-888285c681d1\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") "
Jul 31 21:37:58.535559 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.535495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls\") pod \"8c8044c4-cc78-4d68-ac3a-888285c681d1\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") "
Jul 31 21:37:58.535559 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.535516    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c8044c4-cc78-4d68-ac3a-888285c681d1-ca-trust-extracted\") pod \"8c8044c4-cc78-4d68-ac3a-888285c681d1\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") "
Jul 31 21:37:58.535559 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.535533    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n49df\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-kube-api-access-n49df\") pod \"8c8044c4-cc78-4d68-ac3a-888285c681d1\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") "
Jul 31 21:37:58.535812 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.535732    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-installation-pull-secrets\") pod \"8c8044c4-cc78-4d68-ac3a-888285c681d1\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") "
Jul 31 21:37:58.535812 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.535774    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-image-registry-private-configuration\") pod \"8c8044c4-cc78-4d68-ac3a-888285c681d1\" (UID: \"8c8044c4-cc78-4d68-ac3a-888285c681d1\") "
Jul 31 21:37:58.535920 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.535848    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8c8044c4-cc78-4d68-ac3a-888285c681d1" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 21:37:58.536075 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.536055    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-trusted-ca\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:37:58.536356 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.536315    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8c8044c4-cc78-4d68-ac3a-888285c681d1" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 21:37:58.538183 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.538119    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8c8044c4-cc78-4d68-ac3a-888285c681d1" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:37:58.538183 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.538166    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8c8044c4-cc78-4d68-ac3a-888285c681d1" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:37:58.538330 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.538241    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8c8044c4-cc78-4d68-ac3a-888285c681d1" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:37:58.538373 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.538327    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-kube-api-access-n49df" (OuterVolumeSpecName: "kube-api-access-n49df") pod "8c8044c4-cc78-4d68-ac3a-888285c681d1" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1"). InnerVolumeSpecName "kube-api-access-n49df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:37:58.538373 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.538362    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8c8044c4-cc78-4d68-ac3a-888285c681d1" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:37:58.543853 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.543832    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8044c4-cc78-4d68-ac3a-888285c681d1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8c8044c4-cc78-4d68-ac3a-888285c681d1" (UID: "8c8044c4-cc78-4d68-ac3a-888285c681d1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 21:37:58.636736 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.636701    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-installation-pull-secrets\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:37:58.636736 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.636731    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c8044c4-cc78-4d68-ac3a-888285c681d1-image-registry-private-configuration\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:37:58.636736 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.636742    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-certificates\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:37:58.636968 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.636751    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-bound-sa-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:37:58.636968 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.636759    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-registry-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:37:58.636968 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.636768    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c8044c4-cc78-4d68-ac3a-888285c681d1-ca-trust-extracted\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:37:58.636968 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.636775    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n49df\" (UniqueName: \"kubernetes.io/projected/8c8044c4-cc78-4d68-ac3a-888285c681d1-kube-api-access-n49df\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:37:58.878971 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.878878    2561 generic.go:356] "Generic (PLEG): container finished" podID="8c8044c4-cc78-4d68-ac3a-888285c681d1" containerID="419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159" exitCode=0
Jul 31 21:37:58.878971 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.878926    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4"
Jul 31 21:37:58.879473 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.878968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4" event={"ID":"8c8044c4-cc78-4d68-ac3a-888285c681d1","Type":"ContainerDied","Data":"419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159"}
Jul 31 21:37:58.879473 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.879024    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86b4d994cc-8wmt4" event={"ID":"8c8044c4-cc78-4d68-ac3a-888285c681d1","Type":"ContainerDied","Data":"0ca366e6f452afc63b439865479ef3b27689f2ee69833e8a8e2df0b304a16240"}
Jul 31 21:37:58.879473 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.879046    2561 scope.go:117] "RemoveContainer" containerID="419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159"
Jul 31 21:37:58.889814 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.889794    2561 scope.go:117] "RemoveContainer" containerID="419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159"
Jul 31 21:37:58.890418 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:37:58.890394    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159\": container with ID starting with 419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159 not found: ID does not exist" containerID="419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159"
Jul 31 21:37:58.890515 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.890427    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159"} err="failed to get container status \"419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159\": rpc error: code = NotFound desc = could not find container \"419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159\": container with ID starting with 419f01e08b705a9985b95171dd8f873120dde68de3752b0d8ebf4e94dc443159 not found: ID does not exist"
Jul 31 21:37:58.899212 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.899188    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86b4d994cc-8wmt4"]
Jul 31 21:37:58.909828 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:37:58.909807    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86b4d994cc-8wmt4"]
Jul 31 21:38:00.226657 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:00.226623    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c8044c4-cc78-4d68-ac3a-888285c681d1" path="/var/lib/kubelet/pods/8c8044c4-cc78-4d68-ac3a-888285c681d1/volumes"
Jul 31 21:38:07.515667 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:07.515636    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:38:07.519491 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:07.519470    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-86b995f864-xcxh9"
Jul 31 21:38:29.508316 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:29.508282    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:29.527433 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:29.527408    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:30.011752 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:30.011725    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:47.577972 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:47.577935    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 21:38:47.578506 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:47.578264    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy" containerID="cri-o://e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e" gracePeriod=600
Jul 31 21:38:47.578506 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:47.578292    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd" gracePeriod=600
Jul 31 21:38:47.578506 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:47.578320    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="thanos-sidecar" containerID="cri-o://7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3" gracePeriod=600
Jul 31 21:38:47.578506 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:47.578338    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy-web" containerID="cri-o://1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a" gracePeriod=600
Jul 31 21:38:47.578506 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:47.578250    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="prometheus" containerID="cri-o://b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda" gracePeriod=600
Jul 31 21:38:47.578506 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:47.578333    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="config-reloader" containerID="cri-o://c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b" gracePeriod=600
Jul 31 21:38:48.061946 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.061914    2561 generic.go:356] "Generic (PLEG): container finished" podID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerID="f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd" exitCode=0
Jul 31 21:38:48.061946 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.061943    2561 generic.go:356] "Generic (PLEG): container finished" podID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerID="e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e" exitCode=0
Jul 31 21:38:48.061946 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.061949    2561 generic.go:356] "Generic (PLEG): container finished" podID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerID="7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3" exitCode=0
Jul 31 21:38:48.061946 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.061955    2561 generic.go:356] "Generic (PLEG): container finished" podID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerID="c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b" exitCode=0
Jul 31 21:38:48.062316 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.061960    2561 generic.go:356] "Generic (PLEG): container finished" podID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerID="b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda" exitCode=0
Jul 31 21:38:48.062316 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.061992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerDied","Data":"f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd"}
Jul 31 21:38:48.062316 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.062111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerDied","Data":"e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e"}
Jul 31 21:38:48.062316 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.062173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerDied","Data":"7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3"}
Jul 31 21:38:48.062316 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.062217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerDied","Data":"c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b"}
Jul 31 21:38:48.062316 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.062260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerDied","Data":"b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda"}
Jul 31 21:38:48.822014 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.821987    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:48.919655 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.919574    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-metrics-client-ca\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.919655 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.919630    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.919862 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.919657    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-trusted-ca-bundle\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.919862 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.919675    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kc8c\" (UniqueName: \"kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-kube-api-access-4kc8c\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.919862 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.919699    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-web-config\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.919862 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.919722    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-serving-certs-ca-bundle\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.919862 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.919751    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-2\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920114 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920067    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 21:38:48.920215 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920125    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 21:38:48.920215 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920184    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-tls-assets\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920215 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920188    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 21:38:48.920374 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920238    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-metrics-client-certs\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920374 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920271    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920374 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920299    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-db\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920374 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920325    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-grpc-tls\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920374 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920354    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config-out\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920395    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-thanos-prometheus-http-client-file\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920429    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-1\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920456    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-kube-rbac-proxy\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920490    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-kubelet-serving-ca-bundle\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920528    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-0\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920574    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920600    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-tls\") pod \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\" (UID: \"4d825328-92c9-45ab-b16a-f4aa8b613a7d\") "
Jul 31 21:38:48.920961 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920260    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 21:38:48.920961 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920850    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-metrics-client-ca\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:48.920961 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920924    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:48.920961 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920939    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:48.920961 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.920955    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:48.921219 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.921096    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 21:38:48.921279 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.921265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 21:38:48.922449 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.922413    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 21:38:48.922569 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.922517    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 21:38:48.922742 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.922692    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-kube-api-access-4kc8c" (OuterVolumeSpecName: "kube-api-access-4kc8c") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "kube-api-access-4kc8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:38:48.924516 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.924491    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:38:48.924637 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.924615    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:38:48.924691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.924656    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:38:48.924691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.924672    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config" (OuterVolumeSpecName: "config") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:38:48.924791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.924758    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:38:48.924845 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.924818    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:38:48.924845 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.924831    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:38:48.924949 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.924916    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:38:48.925635 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.925614    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config-out" (OuterVolumeSpecName: "config-out") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 21:38:48.926507 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.926482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:38:48.934562 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:48.934538    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-web-config" (OuterVolumeSpecName: "web-config") pod "4d825328-92c9-45ab-b16a-f4aa8b613a7d" (UID: "4d825328-92c9-45ab-b16a-f4aa8b613a7d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:38:49.022097 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022064    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022097 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022093    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kc8c\" (UniqueName: \"kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-kube-api-access-4kc8c\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022106    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-web-config\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022114    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d825328-92c9-45ab-b16a-f4aa8b613a7d-tls-assets\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022122    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-metrics-client-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022147    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022156    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-db\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022164    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-grpc-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022173    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d825328-92c9-45ab-b16a-f4aa8b613a7d-config-out\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022180    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022189    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022197    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-kube-rbac-proxy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022205    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022214    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4d825328-92c9-45ab-b16a-f4aa8b613a7d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022222    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.022294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.022230    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4d825328-92c9-45ab-b16a-f4aa8b613a7d-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:38:49.068904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.068875    2561 generic.go:356] "Generic (PLEG): container finished" podID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerID="1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a" exitCode=0
Jul 31 21:38:49.069019 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.068933    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerDied","Data":"1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a"}
Jul 31 21:38:49.069019 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.068956    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.069019 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.068989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d825328-92c9-45ab-b16a-f4aa8b613a7d","Type":"ContainerDied","Data":"937388336bf30370dbe189f9206840f951a76b7c25c84855b6c50c2af1c2e923"}
Jul 31 21:38:49.069115 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.069022    2561 scope.go:117] "RemoveContainer" containerID="f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd"
Jul 31 21:38:49.076655 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.076636    2561 scope.go:117] "RemoveContainer" containerID="e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e"
Jul 31 21:38:49.083770 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.083750    2561 scope.go:117] "RemoveContainer" containerID="1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a"
Jul 31 21:38:49.090344 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.090319    2561 scope.go:117] "RemoveContainer" containerID="7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3"
Jul 31 21:38:49.092556 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.092510    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 21:38:49.096670 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.096652    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 21:38:49.097693 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.097680    2561 scope.go:117] "RemoveContainer" containerID="c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b"
Jul 31 21:38:49.104756 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.104738    2561 scope.go:117] "RemoveContainer" containerID="b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda"
Jul 31 21:38:49.111582 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.111571    2561 scope.go:117] "RemoveContainer" containerID="866ae0a75d1c5ffb5e5d382b59c1f8f17a0696f5a443e2eae43a0061112b3d28"
Jul 31 21:38:49.117966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.117947    2561 scope.go:117] "RemoveContainer" containerID="f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd"
Jul 31 21:38:49.118217 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:38:49.118199    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd\": container with ID starting with f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd not found: ID does not exist" containerID="f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd"
Jul 31 21:38:49.118264 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.118225    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd"} err="failed to get container status \"f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd\": rpc error: code = NotFound desc = could not find container \"f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd\": container with ID starting with f1eae8c4c332796de8613933df75aee06ba2b2bd8eaa9d9873373a4da159bebd not found: ID does not exist"
Jul 31 21:38:49.118264 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.118245    2561 scope.go:117] "RemoveContainer" containerID="e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e"
Jul 31 21:38:49.118493 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:38:49.118476    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e\": container with ID starting with e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e not found: ID does not exist" containerID="e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e"
Jul 31 21:38:49.118536 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.118501    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e"} err="failed to get container status \"e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e\": rpc error: code = NotFound desc = could not find container \"e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e\": container with ID starting with e615d38c56c0fe6f2e769fcaddb1da373abc7201191a10f3a89f740c54b85f4e not found: ID does not exist"
Jul 31 21:38:49.118536 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.118518    2561 scope.go:117] "RemoveContainer" containerID="1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a"
Jul 31 21:38:49.118751 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:38:49.118735    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a\": container with ID starting with 1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a not found: ID does not exist" containerID="1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a"
Jul 31 21:38:49.118791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.118757    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a"} err="failed to get container status \"1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a\": rpc error: code = NotFound desc = could not find container \"1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a\": container with ID starting with 1e7f815a035cb6cf2d9ff575d9ce64ec1e1e4b33c28b0f855ce7019943ad926a not found: ID does not exist"
Jul 31 21:38:49.118791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.118772    2561 scope.go:117] "RemoveContainer" containerID="7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3"
Jul 31 21:38:49.118989 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:38:49.118974    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3\": container with ID starting with 7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3 not found: ID does not exist" containerID="7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3"
Jul 31 21:38:49.119034 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.119000    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3"} err="failed to get container status \"7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3\": rpc error: code = NotFound desc = could not find container \"7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3\": container with ID starting with 7d6167e52995bd1369e11d52299525abf08d6a117e2a7ef924c25c3eeccb66a3 not found: ID does not exist"
Jul 31 21:38:49.119034 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.119015    2561 scope.go:117] "RemoveContainer" containerID="c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b"
Jul 31 21:38:49.119334 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:38:49.119317    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b\": container with ID starting with c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b not found: ID does not exist" containerID="c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b"
Jul 31 21:38:49.119391 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.119340    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b"} err="failed to get container status \"c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b\": rpc error: code = NotFound desc = could not find container \"c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b\": container with ID starting with c0a080c45fb86ca407fc9c60e4e9081b8cb3bffc4cf3cf22d3779b91d6c2664b not found: ID does not exist"
Jul 31 21:38:49.119391 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.119359    2561 scope.go:117] "RemoveContainer" containerID="b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda"
Jul 31 21:38:49.119604 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:38:49.119587    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda\": container with ID starting with b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda not found: ID does not exist" containerID="b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda"
Jul 31 21:38:49.119664 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.119611    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda"} err="failed to get container status \"b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda\": rpc error: code = NotFound desc = could not find container \"b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda\": container with ID starting with b4d1aea484e9c7579e25061dc82c47d3f6bc5b3b305227cf8836d02326207dda not found: ID does not exist"
Jul 31 21:38:49.119664 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.119631    2561 scope.go:117] "RemoveContainer" containerID="866ae0a75d1c5ffb5e5d382b59c1f8f17a0696f5a443e2eae43a0061112b3d28"
Jul 31 21:38:49.119840 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:38:49.119824    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866ae0a75d1c5ffb5e5d382b59c1f8f17a0696f5a443e2eae43a0061112b3d28\": container with ID starting with 866ae0a75d1c5ffb5e5d382b59c1f8f17a0696f5a443e2eae43a0061112b3d28 not found: ID does not exist" containerID="866ae0a75d1c5ffb5e5d382b59c1f8f17a0696f5a443e2eae43a0061112b3d28"
Jul 31 21:38:49.119886 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.119843    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866ae0a75d1c5ffb5e5d382b59c1f8f17a0696f5a443e2eae43a0061112b3d28"} err="failed to get container status \"866ae0a75d1c5ffb5e5d382b59c1f8f17a0696f5a443e2eae43a0061112b3d28\": rpc error: code = NotFound desc = could not find container \"866ae0a75d1c5ffb5e5d382b59c1f8f17a0696f5a443e2eae43a0061112b3d28\": container with ID starting with 866ae0a75d1c5ffb5e5d382b59c1f8f17a0696f5a443e2eae43a0061112b3d28 not found: ID does not exist"
Jul 31 21:38:49.123431 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123413    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 21:38:49.123679 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123654    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="init-config-reloader"
Jul 31 21:38:49.123679 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123667    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="init-config-reloader"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123680    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8044c4-cc78-4d68-ac3a-888285c681d1" containerName="registry"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123686    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8044c4-cc78-4d68-ac3a-888285c681d1" containerName="registry"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123695    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="thanos-sidecar"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123701    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="thanos-sidecar"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123708    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy-thanos"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123713    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy-thanos"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123720    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy-web"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123726    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy-web"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123733    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="config-reloader"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123738    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="config-reloader"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123745    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123750    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123759    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="prometheus"
Jul 31 21:38:49.123761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123765    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="prometheus"
Jul 31 21:38:49.124166 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123805    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="prometheus"
Jul 31 21:38:49.124166 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123811    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy-web"
Jul 31 21:38:49.124166 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123816    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy-thanos"
Jul 31 21:38:49.124166 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123823    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c8044c4-cc78-4d68-ac3a-888285c681d1" containerName="registry"
Jul 31 21:38:49.124166 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123851    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="config-reloader"
Jul 31 21:38:49.124166 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123856    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="kube-rbac-proxy"
Jul 31 21:38:49.124166 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.123862    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" containerName="thanos-sidecar"
Jul 31 21:38:49.128802 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.128788    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.134496 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.134475    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 31 21:38:49.134576 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.134475    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5b7qt98vjohdq\""
Jul 31 21:38:49.134687 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.134663    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-p95rf\""
Jul 31 21:38:49.135184 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.135087    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 31 21:38:49.135309 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.135270    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 31 21:38:49.135403 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.135320    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 31 21:38:49.135403 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.135346    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 31 21:38:49.135518 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.135415    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 31 21:38:49.135518 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.135509    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 31 21:38:49.135698 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.135550    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 31 21:38:49.135698 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.135555    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 31 21:38:49.135698 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.135662    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 31 21:38:49.135856 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.135783    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 31 21:38:49.136485 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.136454    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 31 21:38:49.138618 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.138600    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 31 21:38:49.141997 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.141981    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 31 21:38:49.145326 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.145307    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 21:38:49.223740 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-config\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.223740 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.223740 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rp7q\" (UniqueName: \"kubernetes.io/projected/9d88fee0-7666-4359-9724-4724c449c3bd-kube-api-access-9rp7q\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.223740 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d88fee0-7666-4359-9724-4724c449c3bd-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.223740 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224003 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223810    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224003 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223847    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d88fee0-7666-4359-9724-4724c449c3bd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224003 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224003 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224003 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224003 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223958    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224205 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.223998    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224205 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.224049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224205 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.224070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224205 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.224086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224205 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.224109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224205 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.224163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224205 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.224185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224398 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.224208    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.224398 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.224230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325330 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325330 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325552 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325365    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325552 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325388    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325552 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325418    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325552 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-config\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325552 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325485    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325552 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325509    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rp7q\" (UniqueName: \"kubernetes.io/projected/9d88fee0-7666-4359-9724-4724c449c3bd-kube-api-access-9rp7q\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325552 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325536    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d88fee0-7666-4359-9724-4724c449c3bd-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325855 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325855 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325855 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d88fee0-7666-4359-9724-4724c449c3bd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325855 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325855 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325855 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325855 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.325855 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325816    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.326300 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.326300 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.326300 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.325934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.326300 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.326225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.326489 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.326388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.326539 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.326486    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.328360 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.326732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.328360 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.327019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.328360 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.327693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.328360 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.327970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.328646 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.328588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-config\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.328817 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.328696    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.328817 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.328784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.329414 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.329090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.329414 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.329374    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d88fee0-7666-4359-9724-4724c449c3bd-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.329572 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.329449    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.329804 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.329749    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d88fee0-7666-4359-9724-4724c449c3bd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.330671 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.330650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d88fee0-7666-4359-9724-4724c449c3bd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.330783 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.330764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.331198 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.331178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.331524 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.331499    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.331601 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.331543    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d88fee0-7666-4359-9724-4724c449c3bd-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.333114 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.333096    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rp7q\" (UniqueName: \"kubernetes.io/projected/9d88fee0-7666-4359-9724-4724c449c3bd-kube-api-access-9rp7q\") pod \"prometheus-k8s-0\" (UID: \"9d88fee0-7666-4359-9724-4724c449c3bd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.438498 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.438462    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:38:49.574032 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:49.573998    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 21:38:49.577600 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:38:49.577573    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d88fee0_7666_4359_9724_4724c449c3bd.slice/crio-bca8063536d3f154458494513b30c78dae5f3daa54dc3361b176110e6a04ce59 WatchSource:0}: Error finding container bca8063536d3f154458494513b30c78dae5f3daa54dc3361b176110e6a04ce59: Status 404 returned error can't find the container with id bca8063536d3f154458494513b30c78dae5f3daa54dc3361b176110e6a04ce59
Jul 31 21:38:50.075220 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:50.075187    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d88fee0-7666-4359-9724-4724c449c3bd" containerID="abee543d3f1d8c4acb3c729f76c3f10a1e70b9b83c32562141d1f1d704d4f7fe" exitCode=0
Jul 31 21:38:50.075637 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:50.075238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d88fee0-7666-4359-9724-4724c449c3bd","Type":"ContainerDied","Data":"abee543d3f1d8c4acb3c729f76c3f10a1e70b9b83c32562141d1f1d704d4f7fe"}
Jul 31 21:38:50.075637 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:50.075317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d88fee0-7666-4359-9724-4724c449c3bd","Type":"ContainerStarted","Data":"bca8063536d3f154458494513b30c78dae5f3daa54dc3361b176110e6a04ce59"}
Jul 31 21:38:50.228232 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:50.228196    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d825328-92c9-45ab-b16a-f4aa8b613a7d" path="/var/lib/kubelet/pods/4d825328-92c9-45ab-b16a-f4aa8b613a7d/volumes"
Jul 31 21:38:51.082351 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:51.082315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d88fee0-7666-4359-9724-4724c449c3bd","Type":"ContainerStarted","Data":"f80339de51e1d3dadddb2a7d64187e0ac209ca7d84dabcae7c8d694d8b6cdb15"}
Jul 31 21:38:51.082699 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:51.082382    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d88fee0-7666-4359-9724-4724c449c3bd","Type":"ContainerStarted","Data":"66ead9188aec89e30c3f6e115909249339c0d75beef440f4b1f9fe707b272359"}
Jul 31 21:38:51.082699 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:51.082407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d88fee0-7666-4359-9724-4724c449c3bd","Type":"ContainerStarted","Data":"80a6f682ddfdf0c66b67bdb2716a1659dfc090cb2c3a52f2fbef2987119cc5e4"}
Jul 31 21:38:51.082699 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:51.082431    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d88fee0-7666-4359-9724-4724c449c3bd","Type":"ContainerStarted","Data":"eb2757a51e465544ad4b707aa88b75803cc44629c8323ebc460dda3a7245784d"}
Jul 31 21:38:51.082699 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:51.082459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d88fee0-7666-4359-9724-4724c449c3bd","Type":"ContainerStarted","Data":"33cbf91075c7502eb006c6339df9d8f8af886ca0fcd9b71db30f5eae642ccc92"}
Jul 31 21:38:51.082699 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:51.082484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d88fee0-7666-4359-9724-4724c449c3bd","Type":"ContainerStarted","Data":"7c817357319c4fb8886c9bd34a8b5dcbfb17eed87cf134f87a7884f3ab81b36f"}
Jul 31 21:38:51.111067 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:51.111005    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.094523068 podStartE2EDuration="2.110976685s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.453617ms" isStatefulPod=false podCreationTimestamp="2026-07-31 21:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:38:51.109371066 +0000 UTC m=+171.372119430" watchObservedRunningTime="2026-07-31 21:38:51.110976685 +0000 UTC m=+171.373725051"
Jul 31 21:38:54.438698 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:38:54.438608    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:39:49.439313 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:39:49.439272    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:39:49.454464 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:39:49.454440    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:39:50.311623 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:39:50.311594    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 21:41:00.160293 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:00.160258    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 21:41:00.165265 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:00.165242    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 21:41:00.177537 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:00.177513    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 21:41:23.615336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.615302    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-75vlm"]
Jul 31 21:41:23.618538 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.618523    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm"
Jul 31 21:41:23.623243 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.623222    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 31 21:41:23.623629 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.623615    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 31 21:41:23.623686 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.623632    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-9p7jf\""
Jul 31 21:41:23.635079 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.635056    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-75vlm"]
Jul 31 21:41:23.672648 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.672617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbspb\" (UniqueName: \"kubernetes.io/projected/46856ee8-f7eb-4dcb-90f2-860318e6c880-kube-api-access-bbspb\") pod \"cert-manager-webhook-6486ffd997-75vlm\" (UID: \"46856ee8-f7eb-4dcb-90f2-860318e6c880\") " pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm"
Jul 31 21:41:23.672803 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.672661    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46856ee8-f7eb-4dcb-90f2-860318e6c880-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-75vlm\" (UID: \"46856ee8-f7eb-4dcb-90f2-860318e6c880\") " pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm"
Jul 31 21:41:23.773854 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.773819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbspb\" (UniqueName: \"kubernetes.io/projected/46856ee8-f7eb-4dcb-90f2-860318e6c880-kube-api-access-bbspb\") pod \"cert-manager-webhook-6486ffd997-75vlm\" (UID: \"46856ee8-f7eb-4dcb-90f2-860318e6c880\") " pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm"
Jul 31 21:41:23.774050 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.773871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46856ee8-f7eb-4dcb-90f2-860318e6c880-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-75vlm\" (UID: \"46856ee8-f7eb-4dcb-90f2-860318e6c880\") " pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm"
Jul 31 21:41:23.786283 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.786249    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbspb\" (UniqueName: \"kubernetes.io/projected/46856ee8-f7eb-4dcb-90f2-860318e6c880-kube-api-access-bbspb\") pod \"cert-manager-webhook-6486ffd997-75vlm\" (UID: \"46856ee8-f7eb-4dcb-90f2-860318e6c880\") " pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm"
Jul 31 21:41:23.795273 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.795248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46856ee8-f7eb-4dcb-90f2-860318e6c880-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-75vlm\" (UID: \"46856ee8-f7eb-4dcb-90f2-860318e6c880\") " pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm"
Jul 31 21:41:23.927628 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:23.927541    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm"
Jul 31 21:41:24.059677 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:24.059641    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-75vlm"]
Jul 31 21:41:24.062654 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:41:24.062626    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46856ee8_f7eb_4dcb_90f2_860318e6c880.slice/crio-f4a8a576fe92ec7ac2a0c6f1f876de2e8c05473d65e76df5a33b811344f0f656 WatchSource:0}: Error finding container f4a8a576fe92ec7ac2a0c6f1f876de2e8c05473d65e76df5a33b811344f0f656: Status 404 returned error can't find the container with id f4a8a576fe92ec7ac2a0c6f1f876de2e8c05473d65e76df5a33b811344f0f656
Jul 31 21:41:24.064985 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:24.064968    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 21:41:24.644373 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:24.644333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm" event={"ID":"46856ee8-f7eb-4dcb-90f2-860318e6c880","Type":"ContainerStarted","Data":"f4a8a576fe92ec7ac2a0c6f1f876de2e8c05473d65e76df5a33b811344f0f656"}
Jul 31 21:41:27.658673 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:27.658637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm" event={"ID":"46856ee8-f7eb-4dcb-90f2-860318e6c880","Type":"ContainerStarted","Data":"e7d8bfb620932aa03ef9ae52e029a5c2e546fccb34099df6648b73906609b570"}
Jul 31 21:41:27.659089 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:27.658704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm"
Jul 31 21:41:27.677923 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:27.677875    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm" podStartSLOduration=1.6108464630000001 podStartE2EDuration="4.677860079s" totalImagesPullingTime="3.067013616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:41:27.676357044 +0000 UTC m=+327.939105407" watchObservedRunningTime="2026-07-31 21:41:27.677860079 +0000 UTC m=+327.940608444"
Jul 31 21:41:33.664150 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:33.664103    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-75vlm"
Jul 31 21:41:36.632847 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.632813    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv"]
Jul 31 21:41:36.636099 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.636081    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv"
Jul 31 21:41:36.638229 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.638209    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 31 21:41:36.638336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.638254    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 31 21:41:36.638699 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.638678    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-ff7ll\""
Jul 31 21:41:36.650558 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.650536    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv"]
Jul 31 21:41:36.785374 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.785339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f125d920-3042-4343-9b77-a348f29a4dc9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tl9nv\" (UID: \"f125d920-3042-4343-9b77-a348f29a4dc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv"
Jul 31 21:41:36.785550 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.785386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqzrz\" (UniqueName: \"kubernetes.io/projected/f125d920-3042-4343-9b77-a348f29a4dc9-kube-api-access-hqzrz\") pod \"openshift-lws-operator-fd8ccff4c-tl9nv\" (UID: \"f125d920-3042-4343-9b77-a348f29a4dc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv"
Jul 31 21:41:36.886661 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.886569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqzrz\" (UniqueName: \"kubernetes.io/projected/f125d920-3042-4343-9b77-a348f29a4dc9-kube-api-access-hqzrz\") pod \"openshift-lws-operator-fd8ccff4c-tl9nv\" (UID: \"f125d920-3042-4343-9b77-a348f29a4dc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv"
Jul 31 21:41:36.886661 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.886654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f125d920-3042-4343-9b77-a348f29a4dc9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tl9nv\" (UID: \"f125d920-3042-4343-9b77-a348f29a4dc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv"
Jul 31 21:41:36.887044 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.887024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f125d920-3042-4343-9b77-a348f29a4dc9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tl9nv\" (UID: \"f125d920-3042-4343-9b77-a348f29a4dc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv"
Jul 31 21:41:36.894328 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.894305    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqzrz\" (UniqueName: \"kubernetes.io/projected/f125d920-3042-4343-9b77-a348f29a4dc9-kube-api-access-hqzrz\") pod \"openshift-lws-operator-fd8ccff4c-tl9nv\" (UID: \"f125d920-3042-4343-9b77-a348f29a4dc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv"
Jul 31 21:41:36.945263 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:36.945232    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv"
Jul 31 21:41:37.080400 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:37.080234    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv"]
Jul 31 21:41:37.083035 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:41:37.083005    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf125d920_3042_4343_9b77_a348f29a4dc9.slice/crio-1da354e030c390faa0dd2cdae6e0a635030b5302d76606d457d956b2478c96db WatchSource:0}: Error finding container 1da354e030c390faa0dd2cdae6e0a635030b5302d76606d457d956b2478c96db: Status 404 returned error can't find the container with id 1da354e030c390faa0dd2cdae6e0a635030b5302d76606d457d956b2478c96db
Jul 31 21:41:37.696304 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:37.696263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv" event={"ID":"f125d920-3042-4343-9b77-a348f29a4dc9","Type":"ContainerStarted","Data":"1da354e030c390faa0dd2cdae6e0a635030b5302d76606d457d956b2478c96db"}
Jul 31 21:41:39.711313 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:39.711209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv" event={"ID":"f125d920-3042-4343-9b77-a348f29a4dc9","Type":"ContainerStarted","Data":"624992c5060b13c20ed3db656c4793d1fffaef93e29021bcc605b018aeb09b4d"}
Jul 31 21:41:39.736740 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:39.736683    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tl9nv" podStartSLOduration=1.360862891 podStartE2EDuration="3.736662621s" totalImagesPullingTime="2.37579973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:41:39.735573488 +0000 UTC m=+339.998321852" watchObservedRunningTime="2026-07-31 21:41:39.736662621 +0000 UTC m=+339.999410986"
Jul 31 21:41:58.159790 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.159708    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"]
Jul 31 21:41:58.162851 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.162828    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.165843 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.165825    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6wpdq\""
Jul 31 21:41:58.165943 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.165838    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 31 21:41:58.166780 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.166764    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 31 21:41:58.166972 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.166959    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 31 21:41:58.167028 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.166986    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 31 21:41:58.186181 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.186160    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"]
Jul 31 21:41:58.255702 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.255667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e837a855-7075-4797-8c7e-a8ac8f2ab514-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.255702 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.255698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e837a855-7075-4797-8c7e-a8ac8f2ab514-tmp\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.255911 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.255720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e837a855-7075-4797-8c7e-a8ac8f2ab514-manifests\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.255911 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.255760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwvdt\" (UniqueName: \"kubernetes.io/projected/e837a855-7075-4797-8c7e-a8ac8f2ab514-kube-api-access-xwvdt\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.255911 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.255789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e837a855-7075-4797-8c7e-a8ac8f2ab514-webhook-cert\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.356848 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.356794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e837a855-7075-4797-8c7e-a8ac8f2ab514-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.356848 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.356850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e837a855-7075-4797-8c7e-a8ac8f2ab514-tmp\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.357118 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.356875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e837a855-7075-4797-8c7e-a8ac8f2ab514-manifests\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.357118 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.356909    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwvdt\" (UniqueName: \"kubernetes.io/projected/e837a855-7075-4797-8c7e-a8ac8f2ab514-kube-api-access-xwvdt\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.357118 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.356932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e837a855-7075-4797-8c7e-a8ac8f2ab514-webhook-cert\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.357468 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.357439    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e837a855-7075-4797-8c7e-a8ac8f2ab514-manifests\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.359260 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.359238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e837a855-7075-4797-8c7e-a8ac8f2ab514-tmp\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.359370 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.359357    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e837a855-7075-4797-8c7e-a8ac8f2ab514-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.359569 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.359548    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e837a855-7075-4797-8c7e-a8ac8f2ab514-webhook-cert\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.368240 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.368214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwvdt\" (UniqueName: \"kubernetes.io/projected/e837a855-7075-4797-8c7e-a8ac8f2ab514-kube-api-access-xwvdt\") pod \"opendatahub-operator-controller-manager-67685b9c4d-gnpfk\" (UID: \"e837a855-7075-4797-8c7e-a8ac8f2ab514\") " pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.473201 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.473114    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:41:58.616708 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.616648    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"]
Jul 31 21:41:58.619053 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:41:58.619026    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode837a855_7075_4797_8c7e_a8ac8f2ab514.slice/crio-646264ad833068200495bebbaad96637ee435906b4215fc53d08eae2f53ab4a7 WatchSource:0}: Error finding container 646264ad833068200495bebbaad96637ee435906b4215fc53d08eae2f53ab4a7: Status 404 returned error can't find the container with id 646264ad833068200495bebbaad96637ee435906b4215fc53d08eae2f53ab4a7
Jul 31 21:41:58.782426 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:41:58.782388    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk" event={"ID":"e837a855-7075-4797-8c7e-a8ac8f2ab514","Type":"ContainerStarted","Data":"646264ad833068200495bebbaad96637ee435906b4215fc53d08eae2f53ab4a7"}
Jul 31 21:42:01.798828 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:01.798738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk" event={"ID":"e837a855-7075-4797-8c7e-a8ac8f2ab514","Type":"ContainerStarted","Data":"b71d4d48aebfa7dd294d6ad53b5fb3b422b4eed37cf1e6520b1dfa61dc8c6ffb"}
Jul 31 21:42:02.804409 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:02.804365    2561 generic.go:356] "Generic (PLEG): container finished" podID="e837a855-7075-4797-8c7e-a8ac8f2ab514" containerID="b71d4d48aebfa7dd294d6ad53b5fb3b422b4eed37cf1e6520b1dfa61dc8c6ffb" exitCode=0
Jul 31 21:42:02.804785 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:02.804436    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk" event={"ID":"e837a855-7075-4797-8c7e-a8ac8f2ab514","Type":"ContainerDied","Data":"b71d4d48aebfa7dd294d6ad53b5fb3b422b4eed37cf1e6520b1dfa61dc8c6ffb"}
Jul 31 21:42:03.812396 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:03.812353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk" event={"ID":"e837a855-7075-4797-8c7e-a8ac8f2ab514","Type":"ContainerStarted","Data":"0785f40ce538e771ab89daa833c6f19add6777caff3d3572f916a473b9b9b93d"}
Jul 31 21:42:03.812794 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:03.812502    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:42:14.819199 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:14.819165    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk"
Jul 31 21:42:14.855453 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:14.855399    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67685b9c4d-gnpfk" podStartSLOduration=12.531443956 podStartE2EDuration="16.85538267s" totalImagesPullingTime="3.244334677s" totalInitContainerRuntime="1.079604037s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:41:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:42:03.844253288 +0000 UTC m=+364.107001652" watchObservedRunningTime="2026-07-31 21:42:14.85538267 +0000 UTC m=+375.118131033"
Jul 31 21:42:15.602395 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.602335    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"]
Jul 31 21:42:15.611492 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.611466    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.618368 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.618164    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5d8wk\""
Jul 31 21:42:15.619700 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.619587    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 31 21:42:15.620397 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.620064    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 31 21:42:15.620397 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.620216    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 31 21:42:15.622305 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.622279    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"]
Jul 31 21:42:15.700042 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.700003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/43e7bdfd-3aef-44a0-b785-20f877f0da2f-manager-config\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.700290 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.700123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43e7bdfd-3aef-44a0-b785-20f877f0da2f-cert\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.700290 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.700206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/43e7bdfd-3aef-44a0-b785-20f877f0da2f-metrics-cert\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.700290 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.700255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bqc\" (UniqueName: \"kubernetes.io/projected/43e7bdfd-3aef-44a0-b785-20f877f0da2f-kube-api-access-h9bqc\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.801587 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.801530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bqc\" (UniqueName: \"kubernetes.io/projected/43e7bdfd-3aef-44a0-b785-20f877f0da2f-kube-api-access-h9bqc\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.801791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.801634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/43e7bdfd-3aef-44a0-b785-20f877f0da2f-manager-config\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.801791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.801702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43e7bdfd-3aef-44a0-b785-20f877f0da2f-cert\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.801791 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.801730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/43e7bdfd-3aef-44a0-b785-20f877f0da2f-metrics-cert\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.802452 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.802426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/43e7bdfd-3aef-44a0-b785-20f877f0da2f-manager-config\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.804296 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.804267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/43e7bdfd-3aef-44a0-b785-20f877f0da2f-metrics-cert\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.804391 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.804323    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43e7bdfd-3aef-44a0-b785-20f877f0da2f-cert\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.809183 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.809164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bqc\" (UniqueName: \"kubernetes.io/projected/43e7bdfd-3aef-44a0-b785-20f877f0da2f-kube-api-access-h9bqc\") pod \"lws-controller-manager-6bf58c775b-5drkm\" (UID: \"43e7bdfd-3aef-44a0-b785-20f877f0da2f\") " pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:15.922159 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:15.922029    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:16.061902 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:16.061869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"]
Jul 31 21:42:16.064884 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:42:16.064849    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e7bdfd_3aef_44a0_b785_20f877f0da2f.slice/crio-38ec1c10074c586ddd1f028612d8fe2e6077e06eb0fd2daffaf263d29af7d480 WatchSource:0}: Error finding container 38ec1c10074c586ddd1f028612d8fe2e6077e06eb0fd2daffaf263d29af7d480: Status 404 returned error can't find the container with id 38ec1c10074c586ddd1f028612d8fe2e6077e06eb0fd2daffaf263d29af7d480
Jul 31 21:42:16.866678 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:16.866635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm" event={"ID":"43e7bdfd-3aef-44a0-b785-20f877f0da2f","Type":"ContainerStarted","Data":"38ec1c10074c586ddd1f028612d8fe2e6077e06eb0fd2daffaf263d29af7d480"}
Jul 31 21:42:18.877168 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:18.877113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm" event={"ID":"43e7bdfd-3aef-44a0-b785-20f877f0da2f","Type":"ContainerStarted","Data":"fee2a7dc4fe064651fac2b27ada7b5733cfcbf5094472ae4ebdd5b0bfae62ca9"}
Jul 31 21:42:18.877560 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:18.877201    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:42:18.931084 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:18.931028    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm" podStartSLOduration=2.147603634 podStartE2EDuration="3.930999262s" totalImagesPullingTime="1.783395628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:42:18.926538094 +0000 UTC m=+379.189286459" watchObservedRunningTime="2026-07-31 21:42:18.930999262 +0000 UTC m=+379.193747625"
Jul 31 21:42:24.658075 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.658040    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"]
Jul 31 21:42:24.661278 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.661258    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"
Jul 31 21:42:24.663824 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.663797    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 31 21:42:24.664653 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.664630    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 31 21:42:24.664766 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.664631    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 31 21:42:24.665598 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.665578    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 31 21:42:24.667506 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.667487    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wvjqj\""
Jul 31 21:42:24.695609 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.695583    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"]
Jul 31 21:42:24.777078 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.777043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bpzb\" (UniqueName: \"kubernetes.io/projected/3963e755-e179-43ca-9d44-0a1c1dd6e0bd-kube-api-access-6bpzb\") pod \"kube-auth-proxy-64667dffd7-bl6qh\" (UID: \"3963e755-e179-43ca-9d44-0a1c1dd6e0bd\") " pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"
Jul 31 21:42:24.777273 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.777088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3963e755-e179-43ca-9d44-0a1c1dd6e0bd-tls-certs\") pod \"kube-auth-proxy-64667dffd7-bl6qh\" (UID: \"3963e755-e179-43ca-9d44-0a1c1dd6e0bd\") " pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"
Jul 31 21:42:24.777273 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.777145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3963e755-e179-43ca-9d44-0a1c1dd6e0bd-tmp\") pod \"kube-auth-proxy-64667dffd7-bl6qh\" (UID: \"3963e755-e179-43ca-9d44-0a1c1dd6e0bd\") " pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"
Jul 31 21:42:24.878077 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.878038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3963e755-e179-43ca-9d44-0a1c1dd6e0bd-tls-certs\") pod \"kube-auth-proxy-64667dffd7-bl6qh\" (UID: \"3963e755-e179-43ca-9d44-0a1c1dd6e0bd\") " pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"
Jul 31 21:42:24.878282 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.878121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3963e755-e179-43ca-9d44-0a1c1dd6e0bd-tmp\") pod \"kube-auth-proxy-64667dffd7-bl6qh\" (UID: \"3963e755-e179-43ca-9d44-0a1c1dd6e0bd\") " pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"
Jul 31 21:42:24.878282 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.878251    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bpzb\" (UniqueName: \"kubernetes.io/projected/3963e755-e179-43ca-9d44-0a1c1dd6e0bd-kube-api-access-6bpzb\") pod \"kube-auth-proxy-64667dffd7-bl6qh\" (UID: \"3963e755-e179-43ca-9d44-0a1c1dd6e0bd\") " pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"
Jul 31 21:42:24.880445 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.880420    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3963e755-e179-43ca-9d44-0a1c1dd6e0bd-tls-certs\") pod \"kube-auth-proxy-64667dffd7-bl6qh\" (UID: \"3963e755-e179-43ca-9d44-0a1c1dd6e0bd\") " pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"
Jul 31 21:42:24.880583 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.880565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3963e755-e179-43ca-9d44-0a1c1dd6e0bd-tmp\") pod \"kube-auth-proxy-64667dffd7-bl6qh\" (UID: \"3963e755-e179-43ca-9d44-0a1c1dd6e0bd\") " pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"
Jul 31 21:42:24.887256 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.887230    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bpzb\" (UniqueName: \"kubernetes.io/projected/3963e755-e179-43ca-9d44-0a1c1dd6e0bd-kube-api-access-6bpzb\") pod \"kube-auth-proxy-64667dffd7-bl6qh\" (UID: \"3963e755-e179-43ca-9d44-0a1c1dd6e0bd\") " pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"
Jul 31 21:42:24.971289 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:24.971200    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"
Jul 31 21:42:25.124821 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:25.124787    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh"]
Jul 31 21:42:25.127892 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:42:25.127865    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3963e755_e179_43ca_9d44_0a1c1dd6e0bd.slice/crio-ac54a68396e88b9b18cc23b8407d0013618a3bed6760bc4d99227fc47fe727d2 WatchSource:0}: Error finding container ac54a68396e88b9b18cc23b8407d0013618a3bed6760bc4d99227fc47fe727d2: Status 404 returned error can't find the container with id ac54a68396e88b9b18cc23b8407d0013618a3bed6760bc4d99227fc47fe727d2
Jul 31 21:42:25.919444 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:25.919406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh" event={"ID":"3963e755-e179-43ca-9d44-0a1c1dd6e0bd","Type":"ContainerStarted","Data":"ac54a68396e88b9b18cc23b8407d0013618a3bed6760bc4d99227fc47fe727d2"}
Jul 31 21:42:28.935079 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:28.935037    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh" event={"ID":"3963e755-e179-43ca-9d44-0a1c1dd6e0bd","Type":"ContainerStarted","Data":"e75d466c4d847eff6353ee8e31f36c814236e58ec2140fa2d1c8c7ebc7d2dd34"}
Jul 31 21:42:28.955696 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:28.955645    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64667dffd7-bl6qh" podStartSLOduration=2.148534165 podStartE2EDuration="4.955628955s" totalImagesPullingTime="2.80709479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:42:28.954413392 +0000 UTC m=+389.217161756" watchObservedRunningTime="2026-07-31 21:42:28.955628955 +0000 UTC m=+389.218377369"
Jul 31 21:42:29.884053 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:42:29.884022    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bf58c775b-5drkm"
Jul 31 21:44:44.964556 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:44.964522    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-69q65"]
Jul 31 21:44:44.967743 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:44.967726    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:44.970601 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:44.970581    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 31 21:44:44.970711 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:44.970656    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 31 21:44:44.971342 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:44.971324    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 31 21:44:44.971436 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:44.971351    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t5g22\""
Jul 31 21:44:44.980765 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:44.980742    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-69q65"]
Jul 31 21:44:45.064212 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.064181    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-69q65"]
Jul 31 21:44:45.089343 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.089309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nh7x\" (UniqueName: \"kubernetes.io/projected/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-kube-api-access-8nh7x\") pod \"limitador-limitador-859c4d96d6-69q65\" (UID: \"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:45.089498 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.089372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-config-file\") pod \"limitador-limitador-859c4d96d6-69q65\" (UID: \"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:45.190064 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.190025    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nh7x\" (UniqueName: \"kubernetes.io/projected/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-kube-api-access-8nh7x\") pod \"limitador-limitador-859c4d96d6-69q65\" (UID: \"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:45.190250 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.190071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-config-file\") pod \"limitador-limitador-859c4d96d6-69q65\" (UID: \"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:45.190670 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.190650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-config-file\") pod \"limitador-limitador-859c4d96d6-69q65\" (UID: \"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:45.198488 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.198463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nh7x\" (UniqueName: \"kubernetes.io/projected/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-kube-api-access-8nh7x\") pod \"limitador-limitador-859c4d96d6-69q65\" (UID: \"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:45.277334 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.277307    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:45.411020 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.410985    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-69q65"]
Jul 31 21:44:45.414513 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:44:45.414486    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc9add8_1257_46c2_8df5_bd3c6a39e3c0.slice/crio-7678fe7d3929032822d04f5832523e55612b51d64e3141196261bf3a71c07fdd WatchSource:0}: Error finding container 7678fe7d3929032822d04f5832523e55612b51d64e3141196261bf3a71c07fdd: Status 404 returned error can't find the container with id 7678fe7d3929032822d04f5832523e55612b51d64e3141196261bf3a71c07fdd
Jul 31 21:44:45.529582 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.529488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65" event={"ID":"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0","Type":"ContainerStarted","Data":"7678fe7d3929032822d04f5832523e55612b51d64e3141196261bf3a71c07fdd"}
Jul 31 21:44:45.971691 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.971606    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lxpzs"]
Jul 31 21:44:45.975998 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.975977    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-lxpzs"
Jul 31 21:44:45.978541 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.978522    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wcjvp\""
Jul 31 21:44:45.985095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:45.985069    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lxpzs"]
Jul 31 21:44:46.098762 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:46.098727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n627w\" (UniqueName: \"kubernetes.io/projected/cccc2330-d134-4ed9-bbde-6707547ccb58-kube-api-access-n627w\") pod \"authorino-c548b5d7d-lxpzs\" (UID: \"cccc2330-d134-4ed9-bbde-6707547ccb58\") " pod="kuadrant-system/authorino-c548b5d7d-lxpzs"
Jul 31 21:44:46.199996 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:46.199946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n627w\" (UniqueName: \"kubernetes.io/projected/cccc2330-d134-4ed9-bbde-6707547ccb58-kube-api-access-n627w\") pod \"authorino-c548b5d7d-lxpzs\" (UID: \"cccc2330-d134-4ed9-bbde-6707547ccb58\") " pod="kuadrant-system/authorino-c548b5d7d-lxpzs"
Jul 31 21:44:46.219651 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:46.219582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n627w\" (UniqueName: \"kubernetes.io/projected/cccc2330-d134-4ed9-bbde-6707547ccb58-kube-api-access-n627w\") pod \"authorino-c548b5d7d-lxpzs\" (UID: \"cccc2330-d134-4ed9-bbde-6707547ccb58\") " pod="kuadrant-system/authorino-c548b5d7d-lxpzs"
Jul 31 21:44:46.286606 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:46.286566    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-lxpzs"
Jul 31 21:44:46.444048 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:46.443909    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lxpzs"]
Jul 31 21:44:46.447163 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:44:46.447111    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcccc2330_d134_4ed9_bbde_6707547ccb58.slice/crio-ba95bfd2afd470140d3f86326d47bc2108aefe53904adcac94f4a28356339b94 WatchSource:0}: Error finding container ba95bfd2afd470140d3f86326d47bc2108aefe53904adcac94f4a28356339b94: Status 404 returned error can't find the container with id ba95bfd2afd470140d3f86326d47bc2108aefe53904adcac94f4a28356339b94
Jul 31 21:44:46.536424 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:46.536388    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-lxpzs" event={"ID":"cccc2330-d134-4ed9-bbde-6707547ccb58","Type":"ContainerStarted","Data":"ba95bfd2afd470140d3f86326d47bc2108aefe53904adcac94f4a28356339b94"}
Jul 31 21:44:48.549095 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:48.549056    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65" event={"ID":"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0","Type":"ContainerStarted","Data":"64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06"}
Jul 31 21:44:48.549546 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:48.549197    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:48.571361 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:48.571302    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65" podStartSLOduration=2.240879344 podStartE2EDuration="4.571281817s" totalImagesPullingTime="2.330402473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:44:48.570561518 +0000 UTC m=+528.833309876" watchObservedRunningTime="2026-07-31 21:44:48.571281817 +0000 UTC m=+528.834030185"
Jul 31 21:44:49.555157 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:49.555095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-lxpzs" event={"ID":"cccc2330-d134-4ed9-bbde-6707547ccb58","Type":"ContainerStarted","Data":"117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839"}
Jul 31 21:44:49.574323 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:49.574279    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-lxpzs" podStartSLOduration=2.1499854369999998 podStartE2EDuration="4.574264783s" totalImagesPullingTime="2.424279346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:44:49.573422624 +0000 UTC m=+529.836170988" watchObservedRunningTime="2026-07-31 21:44:49.574264783 +0000 UTC m=+529.837013160"
Jul 31 21:44:58.569638 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:58.569559    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-69q65"]
Jul 31 21:44:58.570190 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:58.569752    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65" podUID="bfc9add8-1257-46c2-8df5-bd3c6a39e3c0" containerName="limitador" containerID="cri-o://64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06" gracePeriod=30
Jul 31 21:44:58.570555 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:58.570462    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:59.106990 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.106964    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:59.221030 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.220937    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nh7x\" (UniqueName: \"kubernetes.io/projected/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-kube-api-access-8nh7x\") pod \"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0\" (UID: \"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0\") "
Jul 31 21:44:59.221030 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.221009    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-config-file\") pod \"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0\" (UID: \"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0\") "
Jul 31 21:44:59.221403 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.221380    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-config-file" (OuterVolumeSpecName: "config-file") pod "bfc9add8-1257-46c2-8df5-bd3c6a39e3c0" (UID: "bfc9add8-1257-46c2-8df5-bd3c6a39e3c0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 21:44:59.223143 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.223107    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-kube-api-access-8nh7x" (OuterVolumeSpecName: "kube-api-access-8nh7x") pod "bfc9add8-1257-46c2-8df5-bd3c6a39e3c0" (UID: "bfc9add8-1257-46c2-8df5-bd3c6a39e3c0"). InnerVolumeSpecName "kube-api-access-8nh7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:44:59.322472 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.322426    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nh7x\" (UniqueName: \"kubernetes.io/projected/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-kube-api-access-8nh7x\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:44:59.322472 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.322467    2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0-config-file\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:44:59.604195 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.604157    2561 generic.go:356] "Generic (PLEG): container finished" podID="bfc9add8-1257-46c2-8df5-bd3c6a39e3c0" containerID="64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06" exitCode=0
Jul 31 21:44:59.604683 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.604213    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65"
Jul 31 21:44:59.604683 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.604247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65" event={"ID":"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0","Type":"ContainerDied","Data":"64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06"}
Jul 31 21:44:59.604683 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.604346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-69q65" event={"ID":"bfc9add8-1257-46c2-8df5-bd3c6a39e3c0","Type":"ContainerDied","Data":"7678fe7d3929032822d04f5832523e55612b51d64e3141196261bf3a71c07fdd"}
Jul 31 21:44:59.604683 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.604379    2561 scope.go:117] "RemoveContainer" containerID="64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06"
Jul 31 21:44:59.613344 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.613330    2561 scope.go:117] "RemoveContainer" containerID="64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06"
Jul 31 21:44:59.613653 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:44:59.613630    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06\": container with ID starting with 64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06 not found: ID does not exist" containerID="64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06"
Jul 31 21:44:59.613716 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.613667    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06"} err="failed to get container status \"64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06\": rpc error: code = NotFound desc = could not find container \"64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06\": container with ID starting with 64d21842ffa9c8ec81bfff5b1e8d9ea2d681bb5465407c710c6d1c55d14c3c06 not found: ID does not exist"
Jul 31 21:44:59.626721 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.626689    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-69q65"]
Jul 31 21:44:59.631036 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:44:59.631002    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-69q65"]
Jul 31 21:45:00.229020 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:00.228986    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfc9add8-1257-46c2-8df5-bd3c6a39e3c0" path="/var/lib/kubelet/pods/bfc9add8-1257-46c2-8df5-bd3c6a39e3c0/volumes"
Jul 31 21:45:12.169205 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.169170    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-gpcmc"]
Jul 31 21:45:12.170061 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.170034    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfc9add8-1257-46c2-8df5-bd3c6a39e3c0" containerName="limitador"
Jul 31 21:45:12.170061 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.170062    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfc9add8-1257-46c2-8df5-bd3c6a39e3c0" containerName="limitador"
Jul 31 21:45:12.170275 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.170246    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfc9add8-1257-46c2-8df5-bd3c6a39e3c0" containerName="limitador"
Jul 31 21:45:12.175623 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.175586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-gpcmc"
Jul 31 21:45:12.179610 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.179587    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-gpcmc"]
Jul 31 21:45:12.238639 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.238606    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-gpcmc"]
Jul 31 21:45:12.238858 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:45:12.238835    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bx9wm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-gpcmc" podUID="83b36738-a7b8-4179-9efc-8ed2ed9cd978"
Jul 31 21:45:12.240186 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.240166    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9wm\" (UniqueName: \"kubernetes.io/projected/83b36738-a7b8-4179-9efc-8ed2ed9cd978-kube-api-access-bx9wm\") pod \"authorino-5dc7ccb5f-gpcmc\" (UID: \"83b36738-a7b8-4179-9efc-8ed2ed9cd978\") " pod="kuadrant-system/authorino-5dc7ccb5f-gpcmc"
Jul 31 21:45:12.273206 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.273169    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-dqlq6"]
Jul 31 21:45:12.276458 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.276438    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-dqlq6"
Jul 31 21:45:12.278761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.278736    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 31 21:45:12.285636 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.285612    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-dqlq6"]
Jul 31 21:45:12.341118 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.341079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzscm\" (UniqueName: \"kubernetes.io/projected/2e868cce-7de8-4cd9-9482-c9e194c32de7-kube-api-access-hzscm\") pod \"authorino-687bd45585-dqlq6\" (UID: \"2e868cce-7de8-4cd9-9482-c9e194c32de7\") " pod="kuadrant-system/authorino-687bd45585-dqlq6"
Jul 31 21:45:12.341296 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.341163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9wm\" (UniqueName: \"kubernetes.io/projected/83b36738-a7b8-4179-9efc-8ed2ed9cd978-kube-api-access-bx9wm\") pod \"authorino-5dc7ccb5f-gpcmc\" (UID: \"83b36738-a7b8-4179-9efc-8ed2ed9cd978\") " pod="kuadrant-system/authorino-5dc7ccb5f-gpcmc"
Jul 31 21:45:12.341296 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.341205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e868cce-7de8-4cd9-9482-c9e194c32de7-tls-cert\") pod \"authorino-687bd45585-dqlq6\" (UID: \"2e868cce-7de8-4cd9-9482-c9e194c32de7\") " pod="kuadrant-system/authorino-687bd45585-dqlq6"
Jul 31 21:45:12.349053 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.349018    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9wm\" (UniqueName: \"kubernetes.io/projected/83b36738-a7b8-4179-9efc-8ed2ed9cd978-kube-api-access-bx9wm\") pod \"authorino-5dc7ccb5f-gpcmc\" (UID: \"83b36738-a7b8-4179-9efc-8ed2ed9cd978\") " pod="kuadrant-system/authorino-5dc7ccb5f-gpcmc"
Jul 31 21:45:12.423184 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.423077    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-dqlq6"]
Jul 31 21:45:12.423371 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:45:12.423350    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hzscm tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-dqlq6" podUID="2e868cce-7de8-4cd9-9482-c9e194c32de7"
Jul 31 21:45:12.441705 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.441677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzscm\" (UniqueName: \"kubernetes.io/projected/2e868cce-7de8-4cd9-9482-c9e194c32de7-kube-api-access-hzscm\") pod \"authorino-687bd45585-dqlq6\" (UID: \"2e868cce-7de8-4cd9-9482-c9e194c32de7\") " pod="kuadrant-system/authorino-687bd45585-dqlq6"
Jul 31 21:45:12.441779 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.441744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e868cce-7de8-4cd9-9482-c9e194c32de7-tls-cert\") pod \"authorino-687bd45585-dqlq6\" (UID: \"2e868cce-7de8-4cd9-9482-c9e194c32de7\") " pod="kuadrant-system/authorino-687bd45585-dqlq6"
Jul 31 21:45:12.444460 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.444433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e868cce-7de8-4cd9-9482-c9e194c32de7-tls-cert\") pod \"authorino-687bd45585-dqlq6\" (UID: \"2e868cce-7de8-4cd9-9482-c9e194c32de7\") " pod="kuadrant-system/authorino-687bd45585-dqlq6"
Jul 31 21:45:12.449404 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.449380    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzscm\" (UniqueName: \"kubernetes.io/projected/2e868cce-7de8-4cd9-9482-c9e194c32de7-kube-api-access-hzscm\") pod \"authorino-687bd45585-dqlq6\" (UID: \"2e868cce-7de8-4cd9-9482-c9e194c32de7\") " pod="kuadrant-system/authorino-687bd45585-dqlq6"
Jul 31 21:45:12.659520 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.659488    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-dqlq6"
Jul 31 21:45:12.659684 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.659488    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-gpcmc"
Jul 31 21:45:12.666640 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.666617    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-dqlq6"
Jul 31 21:45:12.669337 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.669316    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-gpcmc"
Jul 31 21:45:12.745052 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.744963    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzscm\" (UniqueName: \"kubernetes.io/projected/2e868cce-7de8-4cd9-9482-c9e194c32de7-kube-api-access-hzscm\") pod \"2e868cce-7de8-4cd9-9482-c9e194c32de7\" (UID: \"2e868cce-7de8-4cd9-9482-c9e194c32de7\") "
Jul 31 21:45:12.745052 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.745016    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx9wm\" (UniqueName: \"kubernetes.io/projected/83b36738-a7b8-4179-9efc-8ed2ed9cd978-kube-api-access-bx9wm\") pod \"83b36738-a7b8-4179-9efc-8ed2ed9cd978\" (UID: \"83b36738-a7b8-4179-9efc-8ed2ed9cd978\") "
Jul 31 21:45:12.745312 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.745106    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e868cce-7de8-4cd9-9482-c9e194c32de7-tls-cert\") pod \"2e868cce-7de8-4cd9-9482-c9e194c32de7\" (UID: \"2e868cce-7de8-4cd9-9482-c9e194c32de7\") "
Jul 31 21:45:12.747177 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.747122    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b36738-a7b8-4179-9efc-8ed2ed9cd978-kube-api-access-bx9wm" (OuterVolumeSpecName: "kube-api-access-bx9wm") pod "83b36738-a7b8-4179-9efc-8ed2ed9cd978" (UID: "83b36738-a7b8-4179-9efc-8ed2ed9cd978"). InnerVolumeSpecName "kube-api-access-bx9wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:45:12.747177 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.747156    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e868cce-7de8-4cd9-9482-c9e194c32de7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2e868cce-7de8-4cd9-9482-c9e194c32de7" (UID: "2e868cce-7de8-4cd9-9482-c9e194c32de7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:45:12.747328 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.747220    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e868cce-7de8-4cd9-9482-c9e194c32de7-kube-api-access-hzscm" (OuterVolumeSpecName: "kube-api-access-hzscm") pod "2e868cce-7de8-4cd9-9482-c9e194c32de7" (UID: "2e868cce-7de8-4cd9-9482-c9e194c32de7"). InnerVolumeSpecName "kube-api-access-hzscm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:45:12.846343 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.846305    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2e868cce-7de8-4cd9-9482-c9e194c32de7-tls-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:45:12.846343 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.846337    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzscm\" (UniqueName: \"kubernetes.io/projected/2e868cce-7de8-4cd9-9482-c9e194c32de7-kube-api-access-hzscm\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:45:12.846343 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:12.846347    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx9wm\" (UniqueName: \"kubernetes.io/projected/83b36738-a7b8-4179-9efc-8ed2ed9cd978-kube-api-access-bx9wm\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:45:13.665063 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:13.665034    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-gpcmc"
Jul 31 21:45:13.665533 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:13.665039    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-dqlq6"
Jul 31 21:45:13.679445 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:13.679419    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lxpzs"]
Jul 31 21:45:13.679598 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:13.679580    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-lxpzs" podUID="cccc2330-d134-4ed9-bbde-6707547ccb58" containerName="authorino" containerID="cri-o://117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839" gracePeriod=30
Jul 31 21:45:13.705212 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:13.705167    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-gpcmc"]
Jul 31 21:45:13.707291 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:13.707267    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-gpcmc"]
Jul 31 21:45:13.727310 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:13.727251    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-dqlq6"]
Jul 31 21:45:13.728904 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:13.728857    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-dqlq6"]
Jul 31 21:45:13.922743 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:13.922688    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-lxpzs"
Jul 31 21:45:13.955622 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:13.955596    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n627w\" (UniqueName: \"kubernetes.io/projected/cccc2330-d134-4ed9-bbde-6707547ccb58-kube-api-access-n627w\") pod \"cccc2330-d134-4ed9-bbde-6707547ccb58\" (UID: \"cccc2330-d134-4ed9-bbde-6707547ccb58\") "
Jul 31 21:45:13.957680 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:13.957653    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccc2330-d134-4ed9-bbde-6707547ccb58-kube-api-access-n627w" (OuterVolumeSpecName: "kube-api-access-n627w") pod "cccc2330-d134-4ed9-bbde-6707547ccb58" (UID: "cccc2330-d134-4ed9-bbde-6707547ccb58"). InnerVolumeSpecName "kube-api-access-n627w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:45:14.056567 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.056532    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n627w\" (UniqueName: \"kubernetes.io/projected/cccc2330-d134-4ed9-bbde-6707547ccb58-kube-api-access-n627w\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:45:14.226876 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.226798    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e868cce-7de8-4cd9-9482-c9e194c32de7" path="/var/lib/kubelet/pods/2e868cce-7de8-4cd9-9482-c9e194c32de7/volumes"
Jul 31 21:45:14.227029 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.227016    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83b36738-a7b8-4179-9efc-8ed2ed9cd978" path="/var/lib/kubelet/pods/83b36738-a7b8-4179-9efc-8ed2ed9cd978/volumes"
Jul 31 21:45:14.669964 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.669931    2561 generic.go:356] "Generic (PLEG): container finished" podID="cccc2330-d134-4ed9-bbde-6707547ccb58" containerID="117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839" exitCode=0
Jul 31 21:45:14.670394 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.669973    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-lxpzs"
Jul 31 21:45:14.670394 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.670023    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-lxpzs" event={"ID":"cccc2330-d134-4ed9-bbde-6707547ccb58","Type":"ContainerDied","Data":"117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839"}
Jul 31 21:45:14.670394 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.670060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-lxpzs" event={"ID":"cccc2330-d134-4ed9-bbde-6707547ccb58","Type":"ContainerDied","Data":"ba95bfd2afd470140d3f86326d47bc2108aefe53904adcac94f4a28356339b94"}
Jul 31 21:45:14.670394 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.670077    2561 scope.go:117] "RemoveContainer" containerID="117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839"
Jul 31 21:45:14.678254 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.678237    2561 scope.go:117] "RemoveContainer" containerID="117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839"
Jul 31 21:45:14.678493 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:45:14.678476    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839\": container with ID starting with 117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839 not found: ID does not exist" containerID="117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839"
Jul 31 21:45:14.678540 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.678502    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839"} err="failed to get container status \"117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839\": rpc error: code = NotFound desc = could not find container \"117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839\": container with ID starting with 117deb3ba0e66784ad74f7e35d82b181c3054b08ee04f77ee36ffa8db519b839 not found: ID does not exist"
Jul 31 21:45:14.685923 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.685901    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lxpzs"]
Jul 31 21:45:14.690068 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:14.690047    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lxpzs"]
Jul 31 21:45:16.226809 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:16.226779    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cccc2330-d134-4ed9-bbde-6707547ccb58" path="/var/lib/kubelet/pods/cccc2330-d134-4ed9-bbde-6707547ccb58/volumes"
Jul 31 21:45:18.375927 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.375891    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8748849bd-rnkfh"]
Jul 31 21:45:18.376478 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.376268    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccc2330-d134-4ed9-bbde-6707547ccb58" containerName="authorino"
Jul 31 21:45:18.376478 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.376281    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccc2330-d134-4ed9-bbde-6707547ccb58" containerName="authorino"
Jul 31 21:45:18.376478 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.376351    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cccc2330-d134-4ed9-bbde-6707547ccb58" containerName="authorino"
Jul 31 21:45:18.380825 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.380804    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:18.384029 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.384007    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 31 21:45:18.384176 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.383982    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 31 21:45:18.384239 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.384221    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-q6jjr\""
Jul 31 21:45:18.396696 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.396672    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8748849bd-rnkfh"]
Jul 31 21:45:18.491924 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.491890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27829546-97f9-4286-86c0-7329d02724e4-webhook-cert\") pod \"maas-controller-8748849bd-rnkfh\" (UID: \"27829546-97f9-4286-86c0-7329d02724e4\") " pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:18.492115 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.491945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpzht\" (UniqueName: \"kubernetes.io/projected/27829546-97f9-4286-86c0-7329d02724e4-kube-api-access-qpzht\") pod \"maas-controller-8748849bd-rnkfh\" (UID: \"27829546-97f9-4286-86c0-7329d02724e4\") " pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:18.541064 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.541029    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79f8b57f55-hfr85"]
Jul 31 21:45:18.544285 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.544263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:45:18.554370 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.554346    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-hfr85"]
Jul 31 21:45:18.592910 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.592873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fcfef25-1044-4d20-9711-129fdc4e3bed-webhook-cert\") pod \"maas-controller-79f8b57f55-hfr85\" (UID: \"1fcfef25-1044-4d20-9711-129fdc4e3bed\") " pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:45:18.593100 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.592949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lk94\" (UniqueName: \"kubernetes.io/projected/1fcfef25-1044-4d20-9711-129fdc4e3bed-kube-api-access-9lk94\") pod \"maas-controller-79f8b57f55-hfr85\" (UID: \"1fcfef25-1044-4d20-9711-129fdc4e3bed\") " pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:45:18.593100 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.593009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27829546-97f9-4286-86c0-7329d02724e4-webhook-cert\") pod \"maas-controller-8748849bd-rnkfh\" (UID: \"27829546-97f9-4286-86c0-7329d02724e4\") " pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:18.593100 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.593074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpzht\" (UniqueName: \"kubernetes.io/projected/27829546-97f9-4286-86c0-7329d02724e4-kube-api-access-qpzht\") pod \"maas-controller-8748849bd-rnkfh\" (UID: \"27829546-97f9-4286-86c0-7329d02724e4\") " pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:18.593261 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:45:18.593191    2561 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 31 21:45:18.593261 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:45:18.593257    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27829546-97f9-4286-86c0-7329d02724e4-webhook-cert podName:27829546-97f9-4286-86c0-7329d02724e4 nodeName:}" failed. No retries permitted until 2026-07-31 21:45:19.093236696 +0000 UTC m=+559.355985054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/27829546-97f9-4286-86c0-7329d02724e4-webhook-cert") pod "maas-controller-8748849bd-rnkfh" (UID: "27829546-97f9-4286-86c0-7329d02724e4") : secret "maas-controller-webhook-cert" not found
Jul 31 21:45:18.602869 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.602839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpzht\" (UniqueName: \"kubernetes.io/projected/27829546-97f9-4286-86c0-7329d02724e4-kube-api-access-qpzht\") pod \"maas-controller-8748849bd-rnkfh\" (UID: \"27829546-97f9-4286-86c0-7329d02724e4\") " pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:18.694564 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.694483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fcfef25-1044-4d20-9711-129fdc4e3bed-webhook-cert\") pod \"maas-controller-79f8b57f55-hfr85\" (UID: \"1fcfef25-1044-4d20-9711-129fdc4e3bed\") " pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:45:18.694564 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.694536    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lk94\" (UniqueName: \"kubernetes.io/projected/1fcfef25-1044-4d20-9711-129fdc4e3bed-kube-api-access-9lk94\") pod \"maas-controller-79f8b57f55-hfr85\" (UID: \"1fcfef25-1044-4d20-9711-129fdc4e3bed\") " pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:45:18.696932 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.696906    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fcfef25-1044-4d20-9711-129fdc4e3bed-webhook-cert\") pod \"maas-controller-79f8b57f55-hfr85\" (UID: \"1fcfef25-1044-4d20-9711-129fdc4e3bed\") " pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:45:18.701661 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.701639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lk94\" (UniqueName: \"kubernetes.io/projected/1fcfef25-1044-4d20-9711-129fdc4e3bed-kube-api-access-9lk94\") pod \"maas-controller-79f8b57f55-hfr85\" (UID: \"1fcfef25-1044-4d20-9711-129fdc4e3bed\") " pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:45:18.855484 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.855435    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:45:18.988479 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:18.988390    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-hfr85"]
Jul 31 21:45:18.992069 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:45:18.992044    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcfef25_1044_4d20_9711_129fdc4e3bed.slice/crio-017fd2ee38bb316a214b45ca3168a1463c531b36a8c1da53793b7ba823e7c594 WatchSource:0}: Error finding container 017fd2ee38bb316a214b45ca3168a1463c531b36a8c1da53793b7ba823e7c594: Status 404 returned error can't find the container with id 017fd2ee38bb316a214b45ca3168a1463c531b36a8c1da53793b7ba823e7c594
Jul 31 21:45:19.098108 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:19.098070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27829546-97f9-4286-86c0-7329d02724e4-webhook-cert\") pod \"maas-controller-8748849bd-rnkfh\" (UID: \"27829546-97f9-4286-86c0-7329d02724e4\") " pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:19.100486 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:19.100466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27829546-97f9-4286-86c0-7329d02724e4-webhook-cert\") pod \"maas-controller-8748849bd-rnkfh\" (UID: \"27829546-97f9-4286-86c0-7329d02724e4\") " pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:19.290832 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:19.290800    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:19.423314 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:19.423290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8748849bd-rnkfh"]
Jul 31 21:45:19.425177 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:45:19.425153    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27829546_97f9_4286_86c0_7329d02724e4.slice/crio-dd7ea79c56dd328e1fae9f94ad278bd5e4d62e794b48a5199a8d60a2c5df11d4 WatchSource:0}: Error finding container dd7ea79c56dd328e1fae9f94ad278bd5e4d62e794b48a5199a8d60a2c5df11d4: Status 404 returned error can't find the container with id dd7ea79c56dd328e1fae9f94ad278bd5e4d62e794b48a5199a8d60a2c5df11d4
Jul 31 21:45:19.699929 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:19.699831    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8b57f55-hfr85" event={"ID":"1fcfef25-1044-4d20-9711-129fdc4e3bed","Type":"ContainerStarted","Data":"017fd2ee38bb316a214b45ca3168a1463c531b36a8c1da53793b7ba823e7c594"}
Jul 31 21:45:19.701114 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:19.701083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8748849bd-rnkfh" event={"ID":"27829546-97f9-4286-86c0-7329d02724e4","Type":"ContainerStarted","Data":"dd7ea79c56dd328e1fae9f94ad278bd5e4d62e794b48a5199a8d60a2c5df11d4"}
Jul 31 21:45:20.707352 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:20.707313    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8b57f55-hfr85" event={"ID":"1fcfef25-1044-4d20-9711-129fdc4e3bed","Type":"ContainerStarted","Data":"402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4"}
Jul 31 21:45:20.707779 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:20.707414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:45:20.708849 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:20.708826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8748849bd-rnkfh" event={"ID":"27829546-97f9-4286-86c0-7329d02724e4","Type":"ContainerStarted","Data":"35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d"}
Jul 31 21:45:20.708963 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:20.708939    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:20.733227 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:20.733175    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79f8b57f55-hfr85" podStartSLOduration=1.166464824 podStartE2EDuration="2.733158239s" totalImagesPullingTime="1.566693415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:45:20.731370941 +0000 UTC m=+560.994119306" watchObservedRunningTime="2026-07-31 21:45:20.733158239 +0000 UTC m=+560.995906602"
Jul 31 21:45:20.756907 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:20.756858    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8748849bd-rnkfh" podStartSLOduration=1.6207240569999999 podStartE2EDuration="2.756841709s" totalImagesPullingTime="1.136117652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:45:20.756022322 +0000 UTC m=+561.018770688" watchObservedRunningTime="2026-07-31 21:45:20.756841709 +0000 UTC m=+561.019590073"
Jul 31 21:45:31.715176 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:31.715123    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:31.715624 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:31.715229    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:45:31.788873 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:31.788832    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8748849bd-rnkfh"]
Jul 31 21:45:31.789076 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:31.789052    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8748849bd-rnkfh" podUID="27829546-97f9-4286-86c0-7329d02724e4" containerName="manager" containerID="cri-o://35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d" gracePeriod=30
Jul 31 21:45:32.037399 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.037376    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:32.111431 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.111395    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27829546-97f9-4286-86c0-7329d02724e4-webhook-cert\") pod \"27829546-97f9-4286-86c0-7329d02724e4\" (UID: \"27829546-97f9-4286-86c0-7329d02724e4\") "
Jul 31 21:45:32.111431 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.111434    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpzht\" (UniqueName: \"kubernetes.io/projected/27829546-97f9-4286-86c0-7329d02724e4-kube-api-access-qpzht\") pod \"27829546-97f9-4286-86c0-7329d02724e4\" (UID: \"27829546-97f9-4286-86c0-7329d02724e4\") "
Jul 31 21:45:32.113684 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.113654    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27829546-97f9-4286-86c0-7329d02724e4-kube-api-access-qpzht" (OuterVolumeSpecName: "kube-api-access-qpzht") pod "27829546-97f9-4286-86c0-7329d02724e4" (UID: "27829546-97f9-4286-86c0-7329d02724e4"). InnerVolumeSpecName "kube-api-access-qpzht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:45:32.113684 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.113671    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27829546-97f9-4286-86c0-7329d02724e4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "27829546-97f9-4286-86c0-7329d02724e4" (UID: "27829546-97f9-4286-86c0-7329d02724e4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:45:32.212827 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.212791    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27829546-97f9-4286-86c0-7329d02724e4-webhook-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:45:32.212827 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.212821    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpzht\" (UniqueName: \"kubernetes.io/projected/27829546-97f9-4286-86c0-7329d02724e4-kube-api-access-qpzht\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:45:32.761997 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.761963    2561 generic.go:356] "Generic (PLEG): container finished" podID="27829546-97f9-4286-86c0-7329d02724e4" containerID="35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d" exitCode=0
Jul 31 21:45:32.762444 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.762003    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8748849bd-rnkfh"
Jul 31 21:45:32.762444 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.762020    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8748849bd-rnkfh" event={"ID":"27829546-97f9-4286-86c0-7329d02724e4","Type":"ContainerDied","Data":"35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d"}
Jul 31 21:45:32.762444 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.762062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8748849bd-rnkfh" event={"ID":"27829546-97f9-4286-86c0-7329d02724e4","Type":"ContainerDied","Data":"dd7ea79c56dd328e1fae9f94ad278bd5e4d62e794b48a5199a8d60a2c5df11d4"}
Jul 31 21:45:32.762444 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.762085    2561 scope.go:117] "RemoveContainer" containerID="35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d"
Jul 31 21:45:32.770094 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.770076    2561 scope.go:117] "RemoveContainer" containerID="35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d"
Jul 31 21:45:32.770353 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:45:32.770336    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d\": container with ID starting with 35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d not found: ID does not exist" containerID="35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d"
Jul 31 21:45:32.770407 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.770360    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d"} err="failed to get container status \"35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d\": rpc error: code = NotFound desc = could not find container \"35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d\": container with ID starting with 35e6bf269302cd52d14d4af0fc270a90228913307acdf6ada6281052a2f5151d not found: ID does not exist"
Jul 31 21:45:32.792021 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.791994    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8748849bd-rnkfh"]
Jul 31 21:45:32.797144 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:32.797109    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8748849bd-rnkfh"]
Jul 31 21:45:34.226889 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:34.226856    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27829546-97f9-4286-86c0-7329d02724e4" path="/var/lib/kubelet/pods/27829546-97f9-4286-86c0-7329d02724e4/volumes"
Jul 31 21:45:55.089231 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.089197    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"]
Jul 31 21:45:55.089692 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.089505    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27829546-97f9-4286-86c0-7329d02724e4" containerName="manager"
Jul 31 21:45:55.089692 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.089516    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27829546-97f9-4286-86c0-7329d02724e4" containerName="manager"
Jul 31 21:45:55.089692 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.089583    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27829546-97f9-4286-86c0-7329d02724e4" containerName="manager"
Jul 31 21:45:55.092396 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.092378    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:55.095281 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.095257    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 31 21:45:55.095424 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.095312    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g8mcz\""
Jul 31 21:45:55.095424 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.095323    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 21:45:55.095424 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.095378    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 21:45:55.104583 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.104558    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"]
Jul 31 21:45:55.118805 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.118782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p8md\" (UniqueName: \"kubernetes.io/projected/a215b518-9422-4eae-8aad-5dcd3fa1d612-kube-api-access-8p8md\") pod \"maas-api-7647b7b95c-2tbzg\" (UID: \"a215b518-9422-4eae-8aad-5dcd3fa1d612\") " pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:55.118937 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.118814    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a215b518-9422-4eae-8aad-5dcd3fa1d612-openshift-service-ca\") pod \"maas-api-7647b7b95c-2tbzg\" (UID: \"a215b518-9422-4eae-8aad-5dcd3fa1d612\") " pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:55.118937 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.118844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a215b518-9422-4eae-8aad-5dcd3fa1d612-maas-api-tls\") pod \"maas-api-7647b7b95c-2tbzg\" (UID: \"a215b518-9422-4eae-8aad-5dcd3fa1d612\") " pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:55.219922 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.219887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p8md\" (UniqueName: \"kubernetes.io/projected/a215b518-9422-4eae-8aad-5dcd3fa1d612-kube-api-access-8p8md\") pod \"maas-api-7647b7b95c-2tbzg\" (UID: \"a215b518-9422-4eae-8aad-5dcd3fa1d612\") " pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:55.219922 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.219927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a215b518-9422-4eae-8aad-5dcd3fa1d612-openshift-service-ca\") pod \"maas-api-7647b7b95c-2tbzg\" (UID: \"a215b518-9422-4eae-8aad-5dcd3fa1d612\") " pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:55.220127 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.219960    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a215b518-9422-4eae-8aad-5dcd3fa1d612-maas-api-tls\") pod \"maas-api-7647b7b95c-2tbzg\" (UID: \"a215b518-9422-4eae-8aad-5dcd3fa1d612\") " pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:55.220670 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.220647    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a215b518-9422-4eae-8aad-5dcd3fa1d612-openshift-service-ca\") pod \"maas-api-7647b7b95c-2tbzg\" (UID: \"a215b518-9422-4eae-8aad-5dcd3fa1d612\") " pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:55.223569 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.223541    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a215b518-9422-4eae-8aad-5dcd3fa1d612-maas-api-tls\") pod \"maas-api-7647b7b95c-2tbzg\" (UID: \"a215b518-9422-4eae-8aad-5dcd3fa1d612\") " pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:55.227936 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.227913    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p8md\" (UniqueName: \"kubernetes.io/projected/a215b518-9422-4eae-8aad-5dcd3fa1d612-kube-api-access-8p8md\") pod \"maas-api-7647b7b95c-2tbzg\" (UID: \"a215b518-9422-4eae-8aad-5dcd3fa1d612\") " pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:55.403024 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.402940    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:55.548294 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.548199    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"]
Jul 31 21:45:55.551117 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:45:55.551082    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda215b518_9422_4eae_8aad_5dcd3fa1d612.slice/crio-dd0e091293ad8358a2cb60b63e3f6d40929becebced0269b33ec7baf49d876e4 WatchSource:0}: Error finding container dd0e091293ad8358a2cb60b63e3f6d40929becebced0269b33ec7baf49d876e4: Status 404 returned error can't find the container with id dd0e091293ad8358a2cb60b63e3f6d40929becebced0269b33ec7baf49d876e4
Jul 31 21:45:55.609472 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.609438    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"]
Jul 31 21:45:55.613825 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.613805    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"
Jul 31 21:45:55.616631 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.616611    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 31 21:45:55.616715 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.616672    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vqnfs\""
Jul 31 21:45:55.622736 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.622533    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrdhm\" (UniqueName: \"kubernetes.io/projected/2b12a163-fd17-4b59-a3a4-c9397968d10a-kube-api-access-hrdhm\") pod \"payload-pre-processing-687dbb6cf6-z4jdf\" (UID: \"2b12a163-fd17-4b59-a3a4-c9397968d10a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"
Jul 31 21:45:55.622854 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.622820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b12a163-fd17-4b59-a3a4-c9397968d10a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-z4jdf\" (UID: \"2b12a163-fd17-4b59-a3a4-c9397968d10a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"
Jul 31 21:45:55.624681 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.624661    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"]
Jul 31 21:45:55.627433 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.627412    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-87xjd"]
Jul 31 21:45:55.630686 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.630665    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd"
Jul 31 21:45:55.644115 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.644090    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-87xjd"]
Jul 31 21:45:55.723572 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.723480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl72d\" (UniqueName: \"kubernetes.io/projected/6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4-kube-api-access-tl72d\") pod \"payload-processing-5cc6cd444b-87xjd\" (UID: \"6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd"
Jul 31 21:45:55.723572 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.723560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrdhm\" (UniqueName: \"kubernetes.io/projected/2b12a163-fd17-4b59-a3a4-c9397968d10a-kube-api-access-hrdhm\") pod \"payload-pre-processing-687dbb6cf6-z4jdf\" (UID: \"2b12a163-fd17-4b59-a3a4-c9397968d10a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"
Jul 31 21:45:55.723799 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.723599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b12a163-fd17-4b59-a3a4-c9397968d10a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-z4jdf\" (UID: \"2b12a163-fd17-4b59-a3a4-c9397968d10a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"
Jul 31 21:45:55.723799 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.723646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-87xjd\" (UID: \"6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd"
Jul 31 21:45:55.724434 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.724411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b12a163-fd17-4b59-a3a4-c9397968d10a-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-z4jdf\" (UID: \"2b12a163-fd17-4b59-a3a4-c9397968d10a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"
Jul 31 21:45:55.732584 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.732563    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrdhm\" (UniqueName: \"kubernetes.io/projected/2b12a163-fd17-4b59-a3a4-c9397968d10a-kube-api-access-hrdhm\") pod \"payload-pre-processing-687dbb6cf6-z4jdf\" (UID: \"2b12a163-fd17-4b59-a3a4-c9397968d10a\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"
Jul 31 21:45:55.824695 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.824654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-87xjd\" (UID: \"6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd"
Jul 31 21:45:55.824867 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.824723    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl72d\" (UniqueName: \"kubernetes.io/projected/6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4-kube-api-access-tl72d\") pod \"payload-processing-5cc6cd444b-87xjd\" (UID: \"6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd"
Jul 31 21:45:55.825481 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.825463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-87xjd\" (UID: \"6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd"
Jul 31 21:45:55.849627 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.849599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl72d\" (UniqueName: \"kubernetes.io/projected/6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4-kube-api-access-tl72d\") pod \"payload-processing-5cc6cd444b-87xjd\" (UID: \"6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd"
Jul 31 21:45:55.860341 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.860312    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg" event={"ID":"a215b518-9422-4eae-8aad-5dcd3fa1d612","Type":"ContainerStarted","Data":"dd0e091293ad8358a2cb60b63e3f6d40929becebced0269b33ec7baf49d876e4"}
Jul 31 21:45:55.924774 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.924740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"
Jul 31 21:45:55.941964 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:55.941933    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd"
Jul 31 21:45:56.069458 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:56.069432    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"]
Jul 31 21:45:56.072386 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:45:56.072351    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b12a163_fd17_4b59_a3a4_c9397968d10a.slice/crio-1af3f16908cb7f215daf72b9d6a9cc08b6e79757b843e1a150fbc1bd7bce3392 WatchSource:0}: Error finding container 1af3f16908cb7f215daf72b9d6a9cc08b6e79757b843e1a150fbc1bd7bce3392: Status 404 returned error can't find the container with id 1af3f16908cb7f215daf72b9d6a9cc08b6e79757b843e1a150fbc1bd7bce3392
Jul 31 21:45:56.096696 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:56.096669    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-87xjd"]
Jul 31 21:45:56.098980 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:45:56.098952    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac7f28e_9e4c_4ff1_8dc3_bd995648ddd4.slice/crio-9b434313953e55ec2db9667a05466afe2aefad01daaf79ccbd253851d9456640 WatchSource:0}: Error finding container 9b434313953e55ec2db9667a05466afe2aefad01daaf79ccbd253851d9456640: Status 404 returned error can't find the container with id 9b434313953e55ec2db9667a05466afe2aefad01daaf79ccbd253851d9456640
Jul 31 21:45:56.869896 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:56.869854    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd" event={"ID":"6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4","Type":"ContainerStarted","Data":"9b434313953e55ec2db9667a05466afe2aefad01daaf79ccbd253851d9456640"}
Jul 31 21:45:56.871450 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:56.871419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf" event={"ID":"2b12a163-fd17-4b59-a3a4-c9397968d10a","Type":"ContainerStarted","Data":"1af3f16908cb7f215daf72b9d6a9cc08b6e79757b843e1a150fbc1bd7bce3392"}
Jul 31 21:45:57.878540 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:57.878453    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg" event={"ID":"a215b518-9422-4eae-8aad-5dcd3fa1d612","Type":"ContainerStarted","Data":"80473e3440cda2f2950d65de34870ba09c9fdf4e1022d833b70c127b2e6ce15e"}
Jul 31 21:45:57.878996 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:57.878604    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:45:57.896146 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:57.896059    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg" podStartSLOduration=0.895331299 podStartE2EDuration="2.896037532s" totalImagesPullingTime="2.000706233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:45:57.895713159 +0000 UTC m=+598.158461525" watchObservedRunningTime="2026-07-31 21:45:57.896037532 +0000 UTC m=+598.158785895"
Jul 31 21:45:58.885219 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:58.885166    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf" event={"ID":"2b12a163-fd17-4b59-a3a4-c9397968d10a","Type":"ContainerStarted","Data":"6f5fe15e07c2a6fc454f9e8f044849e9d58810e9a59567985915f59dcbb79914"}
Jul 31 21:45:58.885690 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:58.885243    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"
Jul 31 21:45:58.886816 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:58.886788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd" event={"ID":"6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4","Type":"ContainerStarted","Data":"957c58829c74cb1d4837dac2d80f906b021f30581fc8acb7c980700ab5b48317"}
Jul 31 21:45:58.886966 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:58.886869    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd"
Jul 31 21:45:58.902005 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:58.901933    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf" podStartSLOduration=1.9261069370000001 podStartE2EDuration="3.90188982s" totalImagesPullingTime="1.975782883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:45:58.899258733 +0000 UTC m=+599.162007098" watchObservedRunningTime="2026-07-31 21:45:58.90188982 +0000 UTC m=+599.164638179"
Jul 31 21:45:58.914914 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:45:58.914870    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd" podStartSLOduration=1.971875935 podStartE2EDuration="3.914854355s" totalImagesPullingTime="1.94297842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:45:58.913411839 +0000 UTC m=+599.176160204" watchObservedRunningTime="2026-07-31 21:45:58.914854355 +0000 UTC m=+599.177602718"
Jul 31 21:46:00.198331 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:00.198295    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 21:46:00.201258 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:00.201230    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 21:46:03.894936 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:03.894901    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7647b7b95c-2tbzg"
Jul 31 21:46:09.893999 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:09.893970    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-z4jdf"
Jul 31 21:46:09.894381 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:09.894033    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-87xjd"
Jul 31 21:46:12.320339 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.320306    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"]
Jul 31 21:46:12.325118 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.325099    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.327250 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.327227    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 31 21:46:12.327779 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.327757    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 31 21:46:12.327883 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.327764    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 31 21:46:12.327883 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.327837    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mxvc5\""
Jul 31 21:46:12.333490 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.333466    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"]
Jul 31 21:46:12.378816 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.378776    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.378816 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.378815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.379021 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.378886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92g5n\" (UniqueName: \"kubernetes.io/projected/3fa59a92-b501-4886-9739-04df669aeb1e-kube-api-access-92g5n\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.379021 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.378948    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.379021 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.378999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa59a92-b501-4886-9739-04df669aeb1e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.379173 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.379021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.479819 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.479783    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.479996 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.479827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.479996 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.479862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92g5n\" (UniqueName: \"kubernetes.io/projected/3fa59a92-b501-4886-9739-04df669aeb1e-kube-api-access-92g5n\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.479996 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.479907    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.479996 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.479962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa59a92-b501-4886-9739-04df669aeb1e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.479996 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.479992    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.480336 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.480306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.480480 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.480427    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.480648 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.480625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.482256 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.482229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa59a92-b501-4886-9739-04df669aeb1e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.482566 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.482546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa59a92-b501-4886-9739-04df669aeb1e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.487371 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.487351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92g5n\" (UniqueName: \"kubernetes.io/projected/3fa59a92-b501-4886-9739-04df669aeb1e-kube-api-access-92g5n\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn\" (UID: \"3fa59a92-b501-4886-9739-04df669aeb1e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.637461 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.637367    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:12.771670 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.771645    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"]
Jul 31 21:46:12.774308 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:46:12.774283    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa59a92_b501_4886_9739_04df669aeb1e.slice/crio-3153358e344698b8ac9107b06681042ea7db00813193acfca5c2621bba4adda5 WatchSource:0}: Error finding container 3153358e344698b8ac9107b06681042ea7db00813193acfca5c2621bba4adda5: Status 404 returned error can't find the container with id 3153358e344698b8ac9107b06681042ea7db00813193acfca5c2621bba4adda5
Jul 31 21:46:12.951823 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:12.951740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn" event={"ID":"3fa59a92-b501-4886-9739-04df669aeb1e","Type":"ContainerStarted","Data":"3153358e344698b8ac9107b06681042ea7db00813193acfca5c2621bba4adda5"}
Jul 31 21:46:13.958255 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:13.958223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn" event={"ID":"3fa59a92-b501-4886-9739-04df669aeb1e","Type":"ContainerStarted","Data":"fca69ffca1ac12d013d2b22bd485ef3b331e17f1eb5e66eedaa9bc36cb802d54"}
Jul 31 21:46:13.980193 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:13.980123    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn" podStartSLOduration=0.960530982 podStartE2EDuration="1.980106365s" totalImagesPullingTime="1.019575383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:46:13.97830486 +0000 UTC m=+614.241053235" watchObservedRunningTime="2026-07-31 21:46:13.980106365 +0000 UTC m=+614.242854728"
Jul 31 21:46:22.638158 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:22.638054    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:22.638158 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:22.638127    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:22.651218 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:22.651190    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:23.009892 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:23.009804    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn"
Jul 31 21:46:28.718386 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.718349    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"]
Jul 31 21:46:28.721961 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.721926    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.725420 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.724212    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 31 21:46:28.729360 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.729336    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"]
Jul 31 21:46:28.826500 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.826455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.826500 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.826500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.826801 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.826541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.826801 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.826597    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g65d\" (UniqueName: \"kubernetes.io/projected/bbf98ded-6e86-4202-9b13-d1b9153e3a22-kube-api-access-5g65d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.826801 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.826629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.826801 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.826654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf98ded-6e86-4202-9b13-d1b9153e3a22-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.927751 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.927716    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.927930 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.927767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.927930 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.927797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.927930 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.927856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g65d\" (UniqueName: \"kubernetes.io/projected/bbf98ded-6e86-4202-9b13-d1b9153e3a22-kube-api-access-5g65d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.927930 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.927883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.927930 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.927914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf98ded-6e86-4202-9b13-d1b9153e3a22-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.928270 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.928240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.928349 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.928294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.928395 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.928376    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.930127 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.930108    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf98ded-6e86-4202-9b13-d1b9153e3a22-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.930388 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.930372    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf98ded-6e86-4202-9b13-d1b9153e3a22-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:28.935489 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:28.935471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g65d\" (UniqueName: \"kubernetes.io/projected/bbf98ded-6e86-4202-9b13-d1b9153e3a22-kube-api-access-5g65d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-8b87k\" (UID: \"bbf98ded-6e86-4202-9b13-d1b9153e3a22\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:29.035075 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:29.035035    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:29.172418 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:29.172391    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"]
Jul 31 21:46:29.174987 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:46:29.174956    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf98ded_6e86_4202_9b13_d1b9153e3a22.slice/crio-18623f631457413cc2097db1d6a1ba21ff914669459b9a84b0649e1286abbc09 WatchSource:0}: Error finding container 18623f631457413cc2097db1d6a1ba21ff914669459b9a84b0649e1286abbc09: Status 404 returned error can't find the container with id 18623f631457413cc2097db1d6a1ba21ff914669459b9a84b0649e1286abbc09
Jul 31 21:46:29.176384 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:29.176365    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 21:46:30.030934 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:30.030887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k" event={"ID":"bbf98ded-6e86-4202-9b13-d1b9153e3a22","Type":"ContainerStarted","Data":"865b8af56c25518f313deaaa13500aa59073941b0736d8e869d998a0ab05cbff"}
Jul 31 21:46:30.031323 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:30.030950    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k" event={"ID":"bbf98ded-6e86-4202-9b13-d1b9153e3a22","Type":"ContainerStarted","Data":"18623f631457413cc2097db1d6a1ba21ff914669459b9a84b0649e1286abbc09"}
Jul 31 21:46:30.074181 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:30.074110    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k" podStartSLOduration=1.884722575 podStartE2EDuration="2.074093382s" totalImagesPullingTime="189.370807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:46:30.072501061 +0000 UTC m=+630.335249444" watchObservedRunningTime="2026-07-31 21:46:30.074093382 +0000 UTC m=+630.336841745"
Jul 31 21:46:39.035886 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:39.035854    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:39.036332 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:39.035945    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:39.048501 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:39.048474    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:46:39.083998 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:46:39.083946    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-8b87k"
Jul 31 21:48:10.975549 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:10.975511    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-678d57fdb7-59l5f"]
Jul 31 21:48:10.978894 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:10.978870    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-678d57fdb7-59l5f"
Jul 31 21:48:10.981283 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:10.981258    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wcjvp\""
Jul 31 21:48:10.981417 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:10.981293    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 31 21:48:10.981417 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:10.981299    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 31 21:48:10.981417 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:10.981325    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 31 21:48:10.986615 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:10.986595    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-678d57fdb7-59l5f"]
Jul 31 21:48:11.138385 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:11.138348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1d36e5c8-8d22-4825-8bdb-a04df6c948f5-tls-cert\") pod \"authorino-678d57fdb7-59l5f\" (UID: \"1d36e5c8-8d22-4825-8bdb-a04df6c948f5\") " pod="kuadrant-system/authorino-678d57fdb7-59l5f"
Jul 31 21:48:11.138385 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:11.138397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srfvk\" (UniqueName: \"kubernetes.io/projected/1d36e5c8-8d22-4825-8bdb-a04df6c948f5-kube-api-access-srfvk\") pod \"authorino-678d57fdb7-59l5f\" (UID: \"1d36e5c8-8d22-4825-8bdb-a04df6c948f5\") " pod="kuadrant-system/authorino-678d57fdb7-59l5f"
Jul 31 21:48:11.239827 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:11.239735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1d36e5c8-8d22-4825-8bdb-a04df6c948f5-tls-cert\") pod \"authorino-678d57fdb7-59l5f\" (UID: \"1d36e5c8-8d22-4825-8bdb-a04df6c948f5\") " pod="kuadrant-system/authorino-678d57fdb7-59l5f"
Jul 31 21:48:11.239827 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:11.239787    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfvk\" (UniqueName: \"kubernetes.io/projected/1d36e5c8-8d22-4825-8bdb-a04df6c948f5-kube-api-access-srfvk\") pod \"authorino-678d57fdb7-59l5f\" (UID: \"1d36e5c8-8d22-4825-8bdb-a04df6c948f5\") " pod="kuadrant-system/authorino-678d57fdb7-59l5f"
Jul 31 21:48:11.242217 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:11.242194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1d36e5c8-8d22-4825-8bdb-a04df6c948f5-tls-cert\") pod \"authorino-678d57fdb7-59l5f\" (UID: \"1d36e5c8-8d22-4825-8bdb-a04df6c948f5\") " pod="kuadrant-system/authorino-678d57fdb7-59l5f"
Jul 31 21:48:11.248817 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:11.248788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srfvk\" (UniqueName: \"kubernetes.io/projected/1d36e5c8-8d22-4825-8bdb-a04df6c948f5-kube-api-access-srfvk\") pod \"authorino-678d57fdb7-59l5f\" (UID: \"1d36e5c8-8d22-4825-8bdb-a04df6c948f5\") " pod="kuadrant-system/authorino-678d57fdb7-59l5f"
Jul 31 21:48:11.289440 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:11.289405    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-678d57fdb7-59l5f"
Jul 31 21:48:11.419755 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:11.419670    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-678d57fdb7-59l5f"]
Jul 31 21:48:11.422440 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:48:11.422410    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d36e5c8_8d22_4825_8bdb_a04df6c948f5.slice/crio-8c3a65de0a41f86742ff9721470c6c6984a51bdc4b7b5fc1e1bf8a1952ee1c76 WatchSource:0}: Error finding container 8c3a65de0a41f86742ff9721470c6c6984a51bdc4b7b5fc1e1bf8a1952ee1c76: Status 404 returned error can't find the container with id 8c3a65de0a41f86742ff9721470c6c6984a51bdc4b7b5fc1e1bf8a1952ee1c76
Jul 31 21:48:11.491811 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:11.491732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-678d57fdb7-59l5f" event={"ID":"1d36e5c8-8d22-4825-8bdb-a04df6c948f5","Type":"ContainerStarted","Data":"8c3a65de0a41f86742ff9721470c6c6984a51bdc4b7b5fc1e1bf8a1952ee1c76"}
Jul 31 21:48:12.498301 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:12.498217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-678d57fdb7-59l5f" event={"ID":"1d36e5c8-8d22-4825-8bdb-a04df6c948f5","Type":"ContainerStarted","Data":"b4b4c1ae645ea1001ec7e9773362330a8aeab926783ba10ac452d945b1d2e8b5"}
Jul 31 21:48:12.514394 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:48:12.514324    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-678d57fdb7-59l5f" podStartSLOduration=1.867427568 podStartE2EDuration="2.514297399s" totalImagesPullingTime="646.869831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:48:12.511623501 +0000 UTC m=+732.774371866" watchObservedRunningTime="2026-07-31 21:48:12.514297399 +0000 UTC m=+732.777045828"
Jul 31 21:49:00.377460 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.377418    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-hfr85"]
Jul 31 21:49:00.378054 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.377669    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-79f8b57f55-hfr85" podUID="1fcfef25-1044-4d20-9711-129fdc4e3bed" containerName="manager" containerID="cri-o://402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4" gracePeriod=30
Jul 31 21:49:00.629699 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.629633    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:49:00.723058 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.723013    2561 generic.go:356] "Generic (PLEG): container finished" podID="1fcfef25-1044-4d20-9711-129fdc4e3bed" containerID="402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4" exitCode=0
Jul 31 21:49:00.723255 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.723071    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8b57f55-hfr85"
Jul 31 21:49:00.723255 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.723090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8b57f55-hfr85" event={"ID":"1fcfef25-1044-4d20-9711-129fdc4e3bed","Type":"ContainerDied","Data":"402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4"}
Jul 31 21:49:00.723255 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.723192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8b57f55-hfr85" event={"ID":"1fcfef25-1044-4d20-9711-129fdc4e3bed","Type":"ContainerDied","Data":"017fd2ee38bb316a214b45ca3168a1463c531b36a8c1da53793b7ba823e7c594"}
Jul 31 21:49:00.723255 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.723218    2561 scope.go:117] "RemoveContainer" containerID="402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4"
Jul 31 21:49:00.731405 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.731386    2561 scope.go:117] "RemoveContainer" containerID="402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4"
Jul 31 21:49:00.731663 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:49:00.731640    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4\": container with ID starting with 402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4 not found: ID does not exist" containerID="402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4"
Jul 31 21:49:00.731727 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.731675    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4"} err="failed to get container status \"402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4\": rpc error: code = NotFound desc = could not find container \"402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4\": container with ID starting with 402805cb4fda1cf41b4f3856d45ddb01f8467a7dd55d2ee781eaf8d9e4f0cee4 not found: ID does not exist"
Jul 31 21:49:00.775037 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.774997    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fcfef25-1044-4d20-9711-129fdc4e3bed-webhook-cert\") pod \"1fcfef25-1044-4d20-9711-129fdc4e3bed\" (UID: \"1fcfef25-1044-4d20-9711-129fdc4e3bed\") "
Jul 31 21:49:00.775200 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.775063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lk94\" (UniqueName: \"kubernetes.io/projected/1fcfef25-1044-4d20-9711-129fdc4e3bed-kube-api-access-9lk94\") pod \"1fcfef25-1044-4d20-9711-129fdc4e3bed\" (UID: \"1fcfef25-1044-4d20-9711-129fdc4e3bed\") "
Jul 31 21:49:00.777127 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.777095    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcfef25-1044-4d20-9711-129fdc4e3bed-kube-api-access-9lk94" (OuterVolumeSpecName: "kube-api-access-9lk94") pod "1fcfef25-1044-4d20-9711-129fdc4e3bed" (UID: "1fcfef25-1044-4d20-9711-129fdc4e3bed"). InnerVolumeSpecName "kube-api-access-9lk94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:49:00.777239 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.777179    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fcfef25-1044-4d20-9711-129fdc4e3bed-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1fcfef25-1044-4d20-9711-129fdc4e3bed" (UID: "1fcfef25-1044-4d20-9711-129fdc4e3bed"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:49:00.875762 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.875725    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fcfef25-1044-4d20-9711-129fdc4e3bed-webhook-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:49:00.875762 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:00.875759    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lk94\" (UniqueName: \"kubernetes.io/projected/1fcfef25-1044-4d20-9711-129fdc4e3bed-kube-api-access-9lk94\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:49:01.046204 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:01.046173    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-hfr85"]
Jul 31 21:49:01.049587 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:01.049563    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-hfr85"]
Jul 31 21:49:02.227121 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:49:02.227089    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fcfef25-1044-4d20-9711-129fdc4e3bed" path="/var/lib/kubelet/pods/1fcfef25-1044-4d20-9711-129fdc4e3bed/volumes"
Jul 31 21:50:01.845581 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:01.845547    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79f8b57f55-gwz8g"]
Jul 31 21:50:01.845983 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:01.845891    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcfef25-1044-4d20-9711-129fdc4e3bed" containerName="manager"
Jul 31 21:50:01.845983 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:01.845902    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcfef25-1044-4d20-9711-129fdc4e3bed" containerName="manager"
Jul 31 21:50:01.845983 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:01.845979    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fcfef25-1044-4d20-9711-129fdc4e3bed" containerName="manager"
Jul 31 21:50:01.851412 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:01.851379    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:01.854314 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:01.854285    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 31 21:50:01.854872 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:01.854834    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 31 21:50:01.854993 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:01.854882    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-q6jjr\""
Jul 31 21:50:01.856253 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:01.856221    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-gwz8g"]
Jul 31 21:50:01.922477 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:01.922433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a2ff9e1-3150-4f74-a0a0-171de04153be-webhook-cert\") pod \"maas-controller-79f8b57f55-gwz8g\" (UID: \"7a2ff9e1-3150-4f74-a0a0-171de04153be\") " pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:01.922667 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:01.922520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlnq7\" (UniqueName: \"kubernetes.io/projected/7a2ff9e1-3150-4f74-a0a0-171de04153be-kube-api-access-nlnq7\") pod \"maas-controller-79f8b57f55-gwz8g\" (UID: \"7a2ff9e1-3150-4f74-a0a0-171de04153be\") " pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:02.023555 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:02.023515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a2ff9e1-3150-4f74-a0a0-171de04153be-webhook-cert\") pod \"maas-controller-79f8b57f55-gwz8g\" (UID: \"7a2ff9e1-3150-4f74-a0a0-171de04153be\") " pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:02.023742 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:02.023566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlnq7\" (UniqueName: \"kubernetes.io/projected/7a2ff9e1-3150-4f74-a0a0-171de04153be-kube-api-access-nlnq7\") pod \"maas-controller-79f8b57f55-gwz8g\" (UID: \"7a2ff9e1-3150-4f74-a0a0-171de04153be\") " pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:02.025993 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:02.025965    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a2ff9e1-3150-4f74-a0a0-171de04153be-webhook-cert\") pod \"maas-controller-79f8b57f55-gwz8g\" (UID: \"7a2ff9e1-3150-4f74-a0a0-171de04153be\") " pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:02.032166 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:02.032125    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlnq7\" (UniqueName: \"kubernetes.io/projected/7a2ff9e1-3150-4f74-a0a0-171de04153be-kube-api-access-nlnq7\") pod \"maas-controller-79f8b57f55-gwz8g\" (UID: \"7a2ff9e1-3150-4f74-a0a0-171de04153be\") " pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:02.164419 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:02.164319    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:02.302034 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:02.301985    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-gwz8g"]
Jul 31 21:50:02.304741 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:50:02.304715    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2ff9e1_3150_4f74_a0a0_171de04153be.slice/crio-9b8a6fdae115f0172bdf7c4aff5443a9a79cc1c7e29486c35897d5ae3aaeff06 WatchSource:0}: Error finding container 9b8a6fdae115f0172bdf7c4aff5443a9a79cc1c7e29486c35897d5ae3aaeff06: Status 404 returned error can't find the container with id 9b8a6fdae115f0172bdf7c4aff5443a9a79cc1c7e29486c35897d5ae3aaeff06
Jul 31 21:50:03.012396 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:03.012360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8b57f55-gwz8g" event={"ID":"7a2ff9e1-3150-4f74-a0a0-171de04153be","Type":"ContainerStarted","Data":"9b8a6fdae115f0172bdf7c4aff5443a9a79cc1c7e29486c35897d5ae3aaeff06"}
Jul 31 21:50:04.018527 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:04.018485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8b57f55-gwz8g" event={"ID":"7a2ff9e1-3150-4f74-a0a0-171de04153be","Type":"ContainerStarted","Data":"5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c"}
Jul 31 21:50:04.018920 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:04.018583    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:04.034779 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:04.034726    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79f8b57f55-gwz8g" podStartSLOduration=2.207341134 podStartE2EDuration="3.03469787s" totalImagesPullingTime="827.356736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:50:04.033903616 +0000 UTC m=+844.296651981" watchObservedRunningTime="2026-07-31 21:50:04.03469787 +0000 UTC m=+844.297446235"
Jul 31 21:50:15.025982 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:15.025951    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:19.642605 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:19.642571    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-gwz8g"]
Jul 31 21:50:19.643062 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:19.642781    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-79f8b57f55-gwz8g" podUID="7a2ff9e1-3150-4f74-a0a0-171de04153be" containerName="manager" containerID="cri-o://5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c" gracePeriod=30
Jul 31 21:50:19.892892 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:19.892838    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:19.982584 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:19.982553    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a2ff9e1-3150-4f74-a0a0-171de04153be-webhook-cert\") pod \"7a2ff9e1-3150-4f74-a0a0-171de04153be\" (UID: \"7a2ff9e1-3150-4f74-a0a0-171de04153be\") "
Jul 31 21:50:19.982771 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:19.982676    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlnq7\" (UniqueName: \"kubernetes.io/projected/7a2ff9e1-3150-4f74-a0a0-171de04153be-kube-api-access-nlnq7\") pod \"7a2ff9e1-3150-4f74-a0a0-171de04153be\" (UID: \"7a2ff9e1-3150-4f74-a0a0-171de04153be\") "
Jul 31 21:50:19.984654 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:19.984628    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2ff9e1-3150-4f74-a0a0-171de04153be-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7a2ff9e1-3150-4f74-a0a0-171de04153be" (UID: "7a2ff9e1-3150-4f74-a0a0-171de04153be"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 21:50:19.984761 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:19.984653    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2ff9e1-3150-4f74-a0a0-171de04153be-kube-api-access-nlnq7" (OuterVolumeSpecName: "kube-api-access-nlnq7") pod "7a2ff9e1-3150-4f74-a0a0-171de04153be" (UID: "7a2ff9e1-3150-4f74-a0a0-171de04153be"). InnerVolumeSpecName "kube-api-access-nlnq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 21:50:20.084280 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.084243    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlnq7\" (UniqueName: \"kubernetes.io/projected/7a2ff9e1-3150-4f74-a0a0-171de04153be-kube-api-access-nlnq7\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:50:20.084280 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.084273    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a2ff9e1-3150-4f74-a0a0-171de04153be-webhook-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 21:50:20.098218 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.098180    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a2ff9e1-3150-4f74-a0a0-171de04153be" containerID="5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c" exitCode=0
Jul 31 21:50:20.098357 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.098232    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8b57f55-gwz8g"
Jul 31 21:50:20.098357 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.098237    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8b57f55-gwz8g" event={"ID":"7a2ff9e1-3150-4f74-a0a0-171de04153be","Type":"ContainerDied","Data":"5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c"}
Jul 31 21:50:20.098357 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.098292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8b57f55-gwz8g" event={"ID":"7a2ff9e1-3150-4f74-a0a0-171de04153be","Type":"ContainerDied","Data":"9b8a6fdae115f0172bdf7c4aff5443a9a79cc1c7e29486c35897d5ae3aaeff06"}
Jul 31 21:50:20.098357 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.098321    2561 scope.go:117] "RemoveContainer" containerID="5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c"
Jul 31 21:50:20.107167 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.107146    2561 scope.go:117] "RemoveContainer" containerID="5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c"
Jul 31 21:50:20.107414 ip-10-0-135-241 kubenswrapper[2561]: E0731 21:50:20.107395    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c\": container with ID starting with 5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c not found: ID does not exist" containerID="5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c"
Jul 31 21:50:20.107465 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.107425    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c"} err="failed to get container status \"5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c\": rpc error: code = NotFound desc = could not find container \"5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c\": container with ID starting with 5a3a1b57ed6f13c7a8bd164d552bff50d76e13ff3d9c560c1583f0a8ba72838c not found: ID does not exist"
Jul 31 21:50:20.116586 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.116562    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-gwz8g"]
Jul 31 21:50:20.119368 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.119346    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-gwz8g"]
Jul 31 21:50:20.228673 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.228593    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a2ff9e1-3150-4f74-a0a0-171de04153be" path="/var/lib/kubelet/pods/7a2ff9e1-3150-4f74-a0a0-171de04153be/volumes"
Jul 31 21:50:20.949187 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.949156    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79f8b57f55-dm5zm"]
Jul 31 21:50:20.949576 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.949562    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2ff9e1-3150-4f74-a0a0-171de04153be" containerName="manager"
Jul 31 21:50:20.949629 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.949578    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2ff9e1-3150-4f74-a0a0-171de04153be" containerName="manager"
Jul 31 21:50:20.949662 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.949640    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2ff9e1-3150-4f74-a0a0-171de04153be" containerName="manager"
Jul 31 21:50:20.953878 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.953861    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8b57f55-dm5zm"
Jul 31 21:50:20.962564 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.962546    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-q6jjr\""
Jul 31 21:50:20.963602 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.963587    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 31 21:50:20.971786 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.971761    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 31 21:50:20.973759 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:20.973735    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-dm5zm"]
Jul 31 21:50:21.092519 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:21.092485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78dfh\" (UniqueName: \"kubernetes.io/projected/a98c3c97-149b-4886-aa32-8708d6f1a775-kube-api-access-78dfh\") pod \"maas-controller-79f8b57f55-dm5zm\" (UID: \"a98c3c97-149b-4886-aa32-8708d6f1a775\") " pod="opendatahub/maas-controller-79f8b57f55-dm5zm"
Jul 31 21:50:21.092698 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:21.092533    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a98c3c97-149b-4886-aa32-8708d6f1a775-webhook-cert\") pod \"maas-controller-79f8b57f55-dm5zm\" (UID: \"a98c3c97-149b-4886-aa32-8708d6f1a775\") " pod="opendatahub/maas-controller-79f8b57f55-dm5zm"
Jul 31 21:50:21.193520 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:21.193481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78dfh\" (UniqueName: \"kubernetes.io/projected/a98c3c97-149b-4886-aa32-8708d6f1a775-kube-api-access-78dfh\") pod \"maas-controller-79f8b57f55-dm5zm\" (UID: \"a98c3c97-149b-4886-aa32-8708d6f1a775\") " pod="opendatahub/maas-controller-79f8b57f55-dm5zm"
Jul 31 21:50:21.193701 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:21.193532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a98c3c97-149b-4886-aa32-8708d6f1a775-webhook-cert\") pod \"maas-controller-79f8b57f55-dm5zm\" (UID: \"a98c3c97-149b-4886-aa32-8708d6f1a775\") " pod="opendatahub/maas-controller-79f8b57f55-dm5zm"
Jul 31 21:50:21.196048 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:21.196025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a98c3c97-149b-4886-aa32-8708d6f1a775-webhook-cert\") pod \"maas-controller-79f8b57f55-dm5zm\" (UID: \"a98c3c97-149b-4886-aa32-8708d6f1a775\") " pod="opendatahub/maas-controller-79f8b57f55-dm5zm"
Jul 31 21:50:21.201360 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:21.201310    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78dfh\" (UniqueName: \"kubernetes.io/projected/a98c3c97-149b-4886-aa32-8708d6f1a775-kube-api-access-78dfh\") pod \"maas-controller-79f8b57f55-dm5zm\" (UID: \"a98c3c97-149b-4886-aa32-8708d6f1a775\") " pod="opendatahub/maas-controller-79f8b57f55-dm5zm"
Jul 31 21:50:21.263632 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:21.263580    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79f8b57f55-dm5zm"
Jul 31 21:50:21.396826 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:21.396801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79f8b57f55-dm5zm"]
Jul 31 21:50:21.399992 ip-10-0-135-241 kubenswrapper[2561]: W0731 21:50:21.399952    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98c3c97_149b_4886_aa32_8708d6f1a775.slice/crio-05410a0bc34066cbed859827cc2bed55e682370157fdf8f869e47a453287a22f WatchSource:0}: Error finding container 05410a0bc34066cbed859827cc2bed55e682370157fdf8f869e47a453287a22f: Status 404 returned error can't find the container with id 05410a0bc34066cbed859827cc2bed55e682370157fdf8f869e47a453287a22f
Jul 31 21:50:22.109985 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:22.109951    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8b57f55-dm5zm" event={"ID":"a98c3c97-149b-4886-aa32-8708d6f1a775","Type":"ContainerStarted","Data":"360231371a45c4e271e5de63842ee3b4ff8a305fe33b3b00fd3b56dcab1316ff"}
Jul 31 21:50:22.110375 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:22.110002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79f8b57f55-dm5zm" event={"ID":"a98c3c97-149b-4886-aa32-8708d6f1a775","Type":"ContainerStarted","Data":"05410a0bc34066cbed859827cc2bed55e682370157fdf8f869e47a453287a22f"}
Jul 31 21:50:22.110375 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:22.110032    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79f8b57f55-dm5zm"
Jul 31 21:50:22.127694 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:22.127597    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79f8b57f55-dm5zm" podStartSLOduration=1.66359138 podStartE2EDuration="2.127580423s" totalImagesPullingTime="463.989043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:50:22.126361381 +0000 UTC m=+862.389109746" watchObservedRunningTime="2026-07-31 21:50:22.127580423 +0000 UTC m=+862.390328787"
Jul 31 21:50:33.117393 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:50:33.117360    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79f8b57f55-dm5zm"
Jul 31 21:51:00.229553 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:51:00.229472    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 21:51:00.230053 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:51:00.230017    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 21:56:00.260052 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:56:00.259967    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 21:56:00.262627 ip-10-0-135-241 kubenswrapper[2561]: I0731 21:56:00.260229    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:00:39.527436 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.527404    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"]
Jul 31 22:00:39.530737 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.530720    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.532759 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.532739    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-463c3cec-kserve-self-signed-certs\""
Jul 31 22:00:39.541978 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.541306    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"]
Jul 31 22:00:39.642934 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.642887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-home\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.642934 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.642944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmrws\" (UniqueName: \"kubernetes.io/projected/a22d11a5-de96-45c5-94fa-1f47dac4abd6-kube-api-access-rmrws\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.643229 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.643009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tmp-dir\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.643229 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.643051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-model-cache\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.643229 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.643075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tls-certs\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.643229 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.643185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-dshm\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.744289 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.744251    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-dshm\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.744490 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.744327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-home\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.744490 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.744363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmrws\" (UniqueName: \"kubernetes.io/projected/a22d11a5-de96-45c5-94fa-1f47dac4abd6-kube-api-access-rmrws\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.744490 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.744394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tmp-dir\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.744490 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.744430    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-model-cache\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.744490 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.744462    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tls-certs\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.744836 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.744813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-home\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.744836 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.744823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tmp-dir\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.744947 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.744829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-model-cache\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.746543 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.746519    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-dshm\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.746784 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.746767    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tls-certs\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.752206 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.752181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmrws\" (UniqueName: \"kubernetes.io/projected/a22d11a5-de96-45c5-94fa-1f47dac4abd6-kube-api-access-rmrws\") pod \"e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") " pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.841738 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.841642    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:39.977128 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.977094    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"]
Jul 31 22:00:39.980574 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:00:39.980550    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda22d11a5_de96_45c5_94fa_1f47dac4abd6.slice/crio-8e3cfffef6b88ca89dfff0cd2cdf4fd68e722937046e236b7c464374c4ff2349 WatchSource:0}: Error finding container 8e3cfffef6b88ca89dfff0cd2cdf4fd68e722937046e236b7c464374c4ff2349: Status 404 returned error can't find the container with id 8e3cfffef6b88ca89dfff0cd2cdf4fd68e722937046e236b7c464374c4ff2349
Jul 31 22:00:39.991679 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:39.991650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99" event={"ID":"a22d11a5-de96-45c5-94fa-1f47dac4abd6","Type":"ContainerStarted","Data":"8e3cfffef6b88ca89dfff0cd2cdf4fd68e722937046e236b7c464374c4ff2349"}
Jul 31 22:00:40.997855 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:40.997816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99" event={"ID":"a22d11a5-de96-45c5-94fa-1f47dac4abd6","Type":"ContainerStarted","Data":"29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620"}
Jul 31 22:00:41.014463 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:41.014414    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99" podStartSLOduration=2.014397614 podStartE2EDuration="2.014397614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:00:41.012631171 +0000 UTC m=+1481.275379535" watchObservedRunningTime="2026-07-31 22:00:41.014397614 +0000 UTC m=+1481.277145978"
Jul 31 22:00:49.313290 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.313258    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"]
Jul 31 22:00:49.317174 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.317152    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.319306 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.319284    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-463c3cec-kserve-self-signed-certs\""
Jul 31 22:00:49.328288 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.328264    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"]
Jul 31 22:00:49.432843 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.432802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-dshm\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.432843 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.432846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpk92\" (UniqueName: \"kubernetes.io/projected/54625e94-2b5b-4496-8bea-1b3e983c7125-kube-api-access-qpk92\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.433051 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.432903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-home\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.433051 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.432937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54625e94-2b5b-4496-8bea-1b3e983c7125-tls-certs\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.433051 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.432966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-model-cache\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.433051 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.432993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-tmp-dir\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.533700 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.533658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-dshm\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.533700 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.533705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpk92\" (UniqueName: \"kubernetes.io/projected/54625e94-2b5b-4496-8bea-1b3e983c7125-kube-api-access-qpk92\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.533963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.533747    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-home\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.533963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.533793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54625e94-2b5b-4496-8bea-1b3e983c7125-tls-certs\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.533963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.533818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-model-cache\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.533963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.533858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-tmp-dir\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.534669 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.534599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-home\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.534802 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.534680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-tmp-dir\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.536225 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.534922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-model-cache\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.536692 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.536666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-dshm\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.542463 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.542433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpk92\" (UniqueName: \"kubernetes.io/projected/54625e94-2b5b-4496-8bea-1b3e983c7125-kube-api-access-qpk92\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.543118 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.543099    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54625e94-2b5b-4496-8bea-1b3e983c7125-tls-certs\") pod \"e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") " pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.630549 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.630461    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:49.766546 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.766518    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"]
Jul 31 22:00:49.768813 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:00:49.768783    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54625e94_2b5b_4496_8bea_1b3e983c7125.slice/crio-ba31cdc64d0d6dd857fdc3e0729adf9a9b5fa8316477cd458ae754ced9e1df27 WatchSource:0}: Error finding container ba31cdc64d0d6dd857fdc3e0729adf9a9b5fa8316477cd458ae754ced9e1df27: Status 404 returned error can't find the container with id ba31cdc64d0d6dd857fdc3e0729adf9a9b5fa8316477cd458ae754ced9e1df27
Jul 31 22:00:49.842695 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.842662    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:49.842907 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.842716    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:49.857440 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:49.857414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:50.051197 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:50.051162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv" event={"ID":"54625e94-2b5b-4496-8bea-1b3e983c7125","Type":"ContainerStarted","Data":"8639065f655c51f6c84e8ebcfee1e75b358839a34a3fa064e2149491369c235b"}
Jul 31 22:00:50.051397 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:50.051207    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv" event={"ID":"54625e94-2b5b-4496-8bea-1b3e983c7125","Type":"ContainerStarted","Data":"ba31cdc64d0d6dd857fdc3e0729adf9a9b5fa8316477cd458ae754ced9e1df27"}
Jul 31 22:00:50.062529 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:50.062496    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:00:50.070155 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:50.070012    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv" podStartSLOduration=1.069994876 podStartE2EDuration="1.069994876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:00:50.069495049 +0000 UTC m=+1490.332243415" watchObservedRunningTime="2026-07-31 22:00:50.069994876 +0000 UTC m=+1490.332743252"
Jul 31 22:00:59.630607 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:59.630571    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:59.631081 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:59.630874    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:00:59.643315 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:00:59.643291    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:01:00.108720 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:00.108689    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:01:00.298151 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:00.298097    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:01:00.300896 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:00.298978    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:01:25.721282 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:25.721187    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"]
Jul 31 22:01:27.060901 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:27.060860    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"]
Jul 31 22:01:40.742349 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:40.742300    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99" podUID="a22d11a5-de96-45c5-94fa-1f47dac4abd6" containerName="main" containerID="cri-o://29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620" gracePeriod=45
Jul 31 22:01:40.992941 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:40.992876    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:01:41.002180 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002156    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-home\") pod \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") "
Jul 31 22:01:41.002266 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002194    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmrws\" (UniqueName: \"kubernetes.io/projected/a22d11a5-de96-45c5-94fa-1f47dac4abd6-kube-api-access-rmrws\") pod \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") "
Jul 31 22:01:41.002266 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002241    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-model-cache\") pod \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") "
Jul 31 22:01:41.002346 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002265    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-dshm\") pod \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") "
Jul 31 22:01:41.002427 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002408    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tmp-dir\") pod \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") "
Jul 31 22:01:41.002481 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002452    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tls-certs\") pod \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\" (UID: \"a22d11a5-de96-45c5-94fa-1f47dac4abd6\") "
Jul 31 22:01:41.002481 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002458    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-home" (OuterVolumeSpecName: "home") pod "a22d11a5-de96-45c5-94fa-1f47dac4abd6" (UID: "a22d11a5-de96-45c5-94fa-1f47dac4abd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:01:41.002557 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002509    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-model-cache" (OuterVolumeSpecName: "model-cache") pod "a22d11a5-de96-45c5-94fa-1f47dac4abd6" (UID: "a22d11a5-de96-45c5-94fa-1f47dac4abd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:01:41.002676 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002648    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a22d11a5-de96-45c5-94fa-1f47dac4abd6" (UID: "a22d11a5-de96-45c5-94fa-1f47dac4abd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:01:41.002804 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002778    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-home\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:41.002804 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002793    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-model-cache\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:41.002804 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.002804    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tmp-dir\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:41.004501 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.004472    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-dshm" (OuterVolumeSpecName: "dshm") pod "a22d11a5-de96-45c5-94fa-1f47dac4abd6" (UID: "a22d11a5-de96-45c5-94fa-1f47dac4abd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:01:41.004589 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.004518    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a22d11a5-de96-45c5-94fa-1f47dac4abd6" (UID: "a22d11a5-de96-45c5-94fa-1f47dac4abd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:01:41.004589 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.004543    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22d11a5-de96-45c5-94fa-1f47dac4abd6-kube-api-access-rmrws" (OuterVolumeSpecName: "kube-api-access-rmrws") pod "a22d11a5-de96-45c5-94fa-1f47dac4abd6" (UID: "a22d11a5-de96-45c5-94fa-1f47dac4abd6"). InnerVolumeSpecName "kube-api-access-rmrws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:01:41.103323 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.103285    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22d11a5-de96-45c5-94fa-1f47dac4abd6-dshm\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:41.103323 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.103317    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22d11a5-de96-45c5-94fa-1f47dac4abd6-tls-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:41.103323 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.103328    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmrws\" (UniqueName: \"kubernetes.io/projected/a22d11a5-de96-45c5-94fa-1f47dac4abd6-kube-api-access-rmrws\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:41.288750 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.288713    2561 generic.go:356] "Generic (PLEG): container finished" podID="a22d11a5-de96-45c5-94fa-1f47dac4abd6" containerID="29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620" exitCode=0
Jul 31 22:01:41.288908 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.288758    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"
Jul 31 22:01:41.288908 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.288780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99" event={"ID":"a22d11a5-de96-45c5-94fa-1f47dac4abd6","Type":"ContainerDied","Data":"29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620"}
Jul 31 22:01:41.288908 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.288819    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99" event={"ID":"a22d11a5-de96-45c5-94fa-1f47dac4abd6","Type":"ContainerDied","Data":"8e3cfffef6b88ca89dfff0cd2cdf4fd68e722937046e236b7c464374c4ff2349"}
Jul 31 22:01:41.288908 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.288840    2561 scope.go:117] "RemoveContainer" containerID="29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620"
Jul 31 22:01:41.298122 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.298101    2561 scope.go:117] "RemoveContainer" containerID="29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620"
Jul 31 22:01:41.298402 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:01:41.298379    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620\": container with ID starting with 29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620 not found: ID does not exist" containerID="29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620"
Jul 31 22:01:41.298454 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.298411    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620"} err="failed to get container status \"29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620\": rpc error: code = NotFound desc = could not find container \"29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620\": container with ID starting with 29ee0ee2569fa03bd665d5dbed6f11d26af97702a38162f40198cb51ad76c620 not found: ID does not exist"
Jul 31 22:01:41.309384 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.309363    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"]
Jul 31 22:01:41.313187 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:41.313167    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-463c3cec-kserve-5f9658df8c-ksf99"]
Jul 31 22:01:42.080096 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.080055    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv" podUID="54625e94-2b5b-4496-8bea-1b3e983c7125" containerName="main" containerID="cri-o://8639065f655c51f6c84e8ebcfee1e75b358839a34a3fa064e2149491369c235b" gracePeriod=45
Jul 31 22:01:42.230129 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.230094    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a22d11a5-de96-45c5-94fa-1f47dac4abd6" path="/var/lib/kubelet/pods/a22d11a5-de96-45c5-94fa-1f47dac4abd6/volumes"
Jul 31 22:01:42.296234 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.296188    2561 generic.go:356] "Generic (PLEG): container finished" podID="54625e94-2b5b-4496-8bea-1b3e983c7125" containerID="8639065f655c51f6c84e8ebcfee1e75b358839a34a3fa064e2149491369c235b" exitCode=0
Jul 31 22:01:42.296416 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.296254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv" event={"ID":"54625e94-2b5b-4496-8bea-1b3e983c7125","Type":"ContainerDied","Data":"8639065f655c51f6c84e8ebcfee1e75b358839a34a3fa064e2149491369c235b"}
Jul 31 22:01:42.336349 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.336288    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:01:42.412289 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpk92\" (UniqueName: \"kubernetes.io/projected/54625e94-2b5b-4496-8bea-1b3e983c7125-kube-api-access-qpk92\") pod \"54625e94-2b5b-4496-8bea-1b3e983c7125\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") "
Jul 31 22:01:42.412289 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412289    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54625e94-2b5b-4496-8bea-1b3e983c7125-tls-certs\") pod \"54625e94-2b5b-4496-8bea-1b3e983c7125\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") "
Jul 31 22:01:42.412479 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412321    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-tmp-dir\") pod \"54625e94-2b5b-4496-8bea-1b3e983c7125\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") "
Jul 31 22:01:42.412479 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412401    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-home\") pod \"54625e94-2b5b-4496-8bea-1b3e983c7125\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") "
Jul 31 22:01:42.412479 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412426    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-model-cache\") pod \"54625e94-2b5b-4496-8bea-1b3e983c7125\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") "
Jul 31 22:01:42.412479 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412457    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-dshm\") pod \"54625e94-2b5b-4496-8bea-1b3e983c7125\" (UID: \"54625e94-2b5b-4496-8bea-1b3e983c7125\") "
Jul 31 22:01:42.412735 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412611    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54625e94-2b5b-4496-8bea-1b3e983c7125" (UID: "54625e94-2b5b-4496-8bea-1b3e983c7125"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:01:42.412735 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412687    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-model-cache" (OuterVolumeSpecName: "model-cache") pod "54625e94-2b5b-4496-8bea-1b3e983c7125" (UID: "54625e94-2b5b-4496-8bea-1b3e983c7125"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:01:42.412735 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412696    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-home" (OuterVolumeSpecName: "home") pod "54625e94-2b5b-4496-8bea-1b3e983c7125" (UID: "54625e94-2b5b-4496-8bea-1b3e983c7125"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:01:42.412954 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412785    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-tmp-dir\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:42.412954 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412804    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-home\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:42.412954 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.412816    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-model-cache\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:42.414831 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.414806    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54625e94-2b5b-4496-8bea-1b3e983c7125-kube-api-access-qpk92" (OuterVolumeSpecName: "kube-api-access-qpk92") pod "54625e94-2b5b-4496-8bea-1b3e983c7125" (UID: "54625e94-2b5b-4496-8bea-1b3e983c7125"). InnerVolumeSpecName "kube-api-access-qpk92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:01:42.414944 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.414852    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54625e94-2b5b-4496-8bea-1b3e983c7125-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54625e94-2b5b-4496-8bea-1b3e983c7125" (UID: "54625e94-2b5b-4496-8bea-1b3e983c7125"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:01:42.415113 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.415089    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-dshm" (OuterVolumeSpecName: "dshm") pod "54625e94-2b5b-4496-8bea-1b3e983c7125" (UID: "54625e94-2b5b-4496-8bea-1b3e983c7125"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:01:42.513692 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.513656    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54625e94-2b5b-4496-8bea-1b3e983c7125-dshm\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:42.513692 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.513689    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpk92\" (UniqueName: \"kubernetes.io/projected/54625e94-2b5b-4496-8bea-1b3e983c7125-kube-api-access-qpk92\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:42.513692 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:42.513699    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54625e94-2b5b-4496-8bea-1b3e983c7125-tls-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:01:43.303208 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:43.303177    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"
Jul 31 22:01:43.303208 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:43.303201    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv" event={"ID":"54625e94-2b5b-4496-8bea-1b3e983c7125","Type":"ContainerDied","Data":"ba31cdc64d0d6dd857fdc3e0729adf9a9b5fa8316477cd458ae754ced9e1df27"}
Jul 31 22:01:43.303726 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:43.303256    2561 scope.go:117] "RemoveContainer" containerID="8639065f655c51f6c84e8ebcfee1e75b358839a34a3fa064e2149491369c235b"
Jul 31 22:01:43.332326 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:43.332294    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"]
Jul 31 22:01:43.341056 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:43.341024    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-463c3cec-kserve-75fdb4d5b7-snlpv"]
Jul 31 22:01:44.228047 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:01:44.228010    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54625e94-2b5b-4496-8bea-1b3e983c7125" path="/var/lib/kubelet/pods/54625e94-2b5b-4496-8bea-1b3e983c7125/volumes"
Jul 31 22:05:26.006145 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.006098    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"]
Jul 31 22:05:26.006614 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.006486    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54625e94-2b5b-4496-8bea-1b3e983c7125" containerName="main"
Jul 31 22:05:26.006614 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.006497    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54625e94-2b5b-4496-8bea-1b3e983c7125" containerName="main"
Jul 31 22:05:26.006614 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.006511    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22d11a5-de96-45c5-94fa-1f47dac4abd6" containerName="main"
Jul 31 22:05:26.006614 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.006516    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22d11a5-de96-45c5-94fa-1f47dac4abd6" containerName="main"
Jul 31 22:05:26.006614 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.006582    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="54625e94-2b5b-4496-8bea-1b3e983c7125" containerName="main"
Jul 31 22:05:26.006614 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.006592    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22d11a5-de96-45c5-94fa-1f47dac4abd6" containerName="main"
Jul 31 22:05:26.009412 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.009393    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.011393 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.011368    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ef199660-serving-cert\""
Jul 31 22:05:26.019900 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.019864    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"]
Jul 31 22:05:26.088556 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.088516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2mf\" (UniqueName: \"kubernetes.io/projected/eb262e18-01e6-474e-9300-3fa4952ecffa-kube-api-access-jb2mf\") pod \"maas-api-e2e-ait-ef199660-577f9f5674-dpmgt\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.088742 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.088576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb262e18-01e6-474e-9300-3fa4952ecffa-openshift-service-ca\") pod \"maas-api-e2e-ait-ef199660-577f9f5674-dpmgt\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.088742 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.088613    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb262e18-01e6-474e-9300-3fa4952ecffa-maas-api-tls\") pod \"maas-api-e2e-ait-ef199660-577f9f5674-dpmgt\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.189604 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.189562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb262e18-01e6-474e-9300-3fa4952ecffa-openshift-service-ca\") pod \"maas-api-e2e-ait-ef199660-577f9f5674-dpmgt\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.189816 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.189629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb262e18-01e6-474e-9300-3fa4952ecffa-maas-api-tls\") pod \"maas-api-e2e-ait-ef199660-577f9f5674-dpmgt\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.189816 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.189696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2mf\" (UniqueName: \"kubernetes.io/projected/eb262e18-01e6-474e-9300-3fa4952ecffa-kube-api-access-jb2mf\") pod \"maas-api-e2e-ait-ef199660-577f9f5674-dpmgt\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.189816 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:05:26.189792    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-serving-cert: secret "maas-api-e2e-ait-ef199660-serving-cert" not found
Jul 31 22:05:26.189978 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:05:26.189873    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb262e18-01e6-474e-9300-3fa4952ecffa-maas-api-tls podName:eb262e18-01e6-474e-9300-3fa4952ecffa nodeName:}" failed. No retries permitted until 2026-07-31 22:05:26.689854312 +0000 UTC m=+1766.952602658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb262e18-01e6-474e-9300-3fa4952ecffa-maas-api-tls") pod "maas-api-e2e-ait-ef199660-577f9f5674-dpmgt" (UID: "eb262e18-01e6-474e-9300-3fa4952ecffa") : secret "maas-api-e2e-ait-ef199660-serving-cert" not found
Jul 31 22:05:26.190253 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.190234    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb262e18-01e6-474e-9300-3fa4952ecffa-openshift-service-ca\") pod \"maas-api-e2e-ait-ef199660-577f9f5674-dpmgt\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.198397 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.198360    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2mf\" (UniqueName: \"kubernetes.io/projected/eb262e18-01e6-474e-9300-3fa4952ecffa-kube-api-access-jb2mf\") pod \"maas-api-e2e-ait-ef199660-577f9f5674-dpmgt\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.637076 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.637038    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"]
Jul 31 22:05:26.640733 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.640713    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:26.643163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.643124    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ef199660\""
Jul 31 22:05:26.651834 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.651805    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"]
Jul 31 22:05:26.676148 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.676103    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"]
Jul 31 22:05:26.679575 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.679553    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:26.681858 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.681832    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ef199660-dockercfg-w2msh\""
Jul 31 22:05:26.690882 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.690841    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"]
Jul 31 22:05:26.694067 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.694037    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrv5\" (UniqueName: \"kubernetes.io/projected/b244ee5b-66d0-4b30-af1b-d443d2f91d44-kube-api-access-9wrv5\") pod \"payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v\" (UID: \"b244ee5b-66d0-4b30-af1b-d443d2f91d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:26.694243 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.694082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb262e18-01e6-474e-9300-3fa4952ecffa-maas-api-tls\") pod \"maas-api-e2e-ait-ef199660-577f9f5674-dpmgt\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.694243 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.694159    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-plugins-config-volume\") pod \"payload-processing-e2e-ait-ef199660-578b77f648-dlktt\" (UID: \"bc5f5b7b-623d-4754-9ecb-c0a06b13863b\") " pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:26.694243 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.694196    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b244ee5b-66d0-4b30-af1b-d443d2f91d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v\" (UID: \"b244ee5b-66d0-4b30-af1b-d443d2f91d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:26.694395 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.694268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjfq7\" (UniqueName: \"kubernetes.io/projected/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-kube-api-access-qjfq7\") pod \"payload-processing-e2e-ait-ef199660-578b77f648-dlktt\" (UID: \"bc5f5b7b-623d-4754-9ecb-c0a06b13863b\") " pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:26.696579 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.696551    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb262e18-01e6-474e-9300-3fa4952ecffa-maas-api-tls\") pod \"maas-api-e2e-ait-ef199660-577f9f5674-dpmgt\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.795324 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.795283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrv5\" (UniqueName: \"kubernetes.io/projected/b244ee5b-66d0-4b30-af1b-d443d2f91d44-kube-api-access-9wrv5\") pod \"payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v\" (UID: \"b244ee5b-66d0-4b30-af1b-d443d2f91d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:26.795508 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.795351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-plugins-config-volume\") pod \"payload-processing-e2e-ait-ef199660-578b77f648-dlktt\" (UID: \"bc5f5b7b-623d-4754-9ecb-c0a06b13863b\") " pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:26.795508 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.795461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b244ee5b-66d0-4b30-af1b-d443d2f91d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v\" (UID: \"b244ee5b-66d0-4b30-af1b-d443d2f91d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:26.795600 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.795515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjfq7\" (UniqueName: \"kubernetes.io/projected/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-kube-api-access-qjfq7\") pod \"payload-processing-e2e-ait-ef199660-578b77f648-dlktt\" (UID: \"bc5f5b7b-623d-4754-9ecb-c0a06b13863b\") " pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:26.796284 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.796262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b244ee5b-66d0-4b30-af1b-d443d2f91d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v\" (UID: \"b244ee5b-66d0-4b30-af1b-d443d2f91d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:26.796331 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.796304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-plugins-config-volume\") pod \"payload-processing-e2e-ait-ef199660-578b77f648-dlktt\" (UID: \"bc5f5b7b-623d-4754-9ecb-c0a06b13863b\") " pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:26.803397 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.803364    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrv5\" (UniqueName: \"kubernetes.io/projected/b244ee5b-66d0-4b30-af1b-d443d2f91d44-kube-api-access-9wrv5\") pod \"payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v\" (UID: \"b244ee5b-66d0-4b30-af1b-d443d2f91d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:26.803653 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.803631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjfq7\" (UniqueName: \"kubernetes.io/projected/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-kube-api-access-qjfq7\") pod \"payload-processing-e2e-ait-ef199660-578b77f648-dlktt\" (UID: \"bc5f5b7b-623d-4754-9ecb-c0a06b13863b\") " pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:26.922105 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.922005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:26.952522 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.952491    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:26.991192 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:26.991042    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:27.092617 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.092586    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"]
Jul 31 22:05:27.096394 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:05:27.096324    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb262e18_01e6_474e_9300_3fa4952ecffa.slice/crio-3931265091b65c933444ec1cb6d32a7d6fb58c4952a5d0ba2e45e4a8fea31bc3 WatchSource:0}: Error finding container 3931265091b65c933444ec1cb6d32a7d6fb58c4952a5d0ba2e45e4a8fea31bc3: Status 404 returned error can't find the container with id 3931265091b65c933444ec1cb6d32a7d6fb58c4952a5d0ba2e45e4a8fea31bc3
Jul 31 22:05:27.098261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.098243    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 22:05:27.132010 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.131919    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"]
Jul 31 22:05:27.134357 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:05:27.134317    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb244ee5b_66d0_4b30_af1b_d443d2f91d44.slice/crio-d8007242aaa48ba21a6f76301c4860982317c7d779cde5001cd2016bc6297b38 WatchSource:0}: Error finding container d8007242aaa48ba21a6f76301c4860982317c7d779cde5001cd2016bc6297b38: Status 404 returned error can't find the container with id d8007242aaa48ba21a6f76301c4860982317c7d779cde5001cd2016bc6297b38
Jul 31 22:05:27.159308 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.159250    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"]
Jul 31 22:05:27.165380 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:05:27.165348    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5f5b7b_623d_4754_9ecb_c0a06b13863b.slice/crio-a5a37a937842139030b8163d7a24e34231dbc4e7edd1253adefcceefe76feab1 WatchSource:0}: Error finding container a5a37a937842139030b8163d7a24e34231dbc4e7edd1253adefcceefe76feab1: Status 404 returned error can't find the container with id a5a37a937842139030b8163d7a24e34231dbc4e7edd1253adefcceefe76feab1
Jul 31 22:05:27.416079 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.416039    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt" event={"ID":"eb262e18-01e6-474e-9300-3fa4952ecffa","Type":"ContainerStarted","Data":"3931265091b65c933444ec1cb6d32a7d6fb58c4952a5d0ba2e45e4a8fea31bc3"}
Jul 31 22:05:27.417716 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.417688    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v" event={"ID":"b244ee5b-66d0-4b30-af1b-d443d2f91d44","Type":"ContainerStarted","Data":"52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb"}
Jul 31 22:05:27.417873 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.417738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v" event={"ID":"b244ee5b-66d0-4b30-af1b-d443d2f91d44","Type":"ContainerStarted","Data":"d8007242aaa48ba21a6f76301c4860982317c7d779cde5001cd2016bc6297b38"}
Jul 31 22:05:27.417873 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.417776    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:27.419283 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.419254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt" event={"ID":"bc5f5b7b-623d-4754-9ecb-c0a06b13863b","Type":"ContainerStarted","Data":"4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3"}
Jul 31 22:05:27.419413 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.419290    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt" event={"ID":"bc5f5b7b-623d-4754-9ecb-c0a06b13863b","Type":"ContainerStarted","Data":"a5a37a937842139030b8163d7a24e34231dbc4e7edd1253adefcceefe76feab1"}
Jul 31 22:05:27.419413 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.419379    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:27.435662 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.435551    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v" podStartSLOduration=1.435532807 podStartE2EDuration="1.435532807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:05:27.434284012 +0000 UTC m=+1767.697032378" watchObservedRunningTime="2026-07-31 22:05:27.435532807 +0000 UTC m=+1767.698281172"
Jul 31 22:05:27.449874 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:27.449806    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt" podStartSLOduration=1.4497855099999999 podStartE2EDuration="1.44978551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:05:27.449310066 +0000 UTC m=+1767.712058439" watchObservedRunningTime="2026-07-31 22:05:27.44978551 +0000 UTC m=+1767.712533874"
Jul 31 22:05:28.426372 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:28.426332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt" event={"ID":"eb262e18-01e6-474e-9300-3fa4952ecffa","Type":"ContainerStarted","Data":"c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49"}
Jul 31 22:05:28.426867 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:28.426462    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:28.441925 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:28.441867    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt" podStartSLOduration=3.003573339 podStartE2EDuration="3.44184624s" totalImagesPullingTime="438.272901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:05:28.4403875 +0000 UTC m=+1768.703135864" watchObservedRunningTime="2026-07-31 22:05:28.44184624 +0000 UTC m=+1768.704594603"
Jul 31 22:05:34.440666 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:34.440633    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:38.428330 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:38.428297    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:38.428823 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:38.428528    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:56.588620 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:56.588553    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"]
Jul 31 22:05:56.589165 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:56.588881    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt" podUID="eb262e18-01e6-474e-9300-3fa4952ecffa" containerName="maas-api" containerID="cri-o://c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49" gracePeriod=30
Jul 31 22:05:56.647835 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:56.647788    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"]
Jul 31 22:05:56.648064 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:56.648036    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt" podUID="bc5f5b7b-623d-4754-9ecb-c0a06b13863b" containerName="payload-processing" containerID="cri-o://4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3" gracePeriod=30
Jul 31 22:05:56.657710 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:56.657676    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"]
Jul 31 22:05:56.657980 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:56.657931    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v" podUID="b244ee5b-66d0-4b30-af1b-d443d2f91d44" containerName="payload-pre-processing" containerID="cri-o://52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb" gracePeriod=30
Jul 31 22:05:56.924081 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:56.924052    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:56.934295 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:56.934268    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:56.967234 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:56.967202    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:57.078962 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.078918    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wrv5\" (UniqueName: \"kubernetes.io/projected/b244ee5b-66d0-4b30-af1b-d443d2f91d44-kube-api-access-9wrv5\") pod \"b244ee5b-66d0-4b30-af1b-d443d2f91d44\" (UID: \"b244ee5b-66d0-4b30-af1b-d443d2f91d44\") "
Jul 31 22:05:57.079283 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.078975    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-plugins-config-volume\") pod \"bc5f5b7b-623d-4754-9ecb-c0a06b13863b\" (UID: \"bc5f5b7b-623d-4754-9ecb-c0a06b13863b\") "
Jul 31 22:05:57.079283 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.079004    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb262e18-01e6-474e-9300-3fa4952ecffa-maas-api-tls\") pod \"eb262e18-01e6-474e-9300-3fa4952ecffa\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") "
Jul 31 22:05:57.079283 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.079036    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjfq7\" (UniqueName: \"kubernetes.io/projected/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-kube-api-access-qjfq7\") pod \"bc5f5b7b-623d-4754-9ecb-c0a06b13863b\" (UID: \"bc5f5b7b-623d-4754-9ecb-c0a06b13863b\") "
Jul 31 22:05:57.079283 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.079060    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b244ee5b-66d0-4b30-af1b-d443d2f91d44-plugins-config-volume\") pod \"b244ee5b-66d0-4b30-af1b-d443d2f91d44\" (UID: \"b244ee5b-66d0-4b30-af1b-d443d2f91d44\") "
Jul 31 22:05:57.079283 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.079165    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb262e18-01e6-474e-9300-3fa4952ecffa-openshift-service-ca\") pod \"eb262e18-01e6-474e-9300-3fa4952ecffa\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") "
Jul 31 22:05:57.079283 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.079223    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb2mf\" (UniqueName: \"kubernetes.io/projected/eb262e18-01e6-474e-9300-3fa4952ecffa-kube-api-access-jb2mf\") pod \"eb262e18-01e6-474e-9300-3fa4952ecffa\" (UID: \"eb262e18-01e6-474e-9300-3fa4952ecffa\") "
Jul 31 22:05:57.079592 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.079524    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b244ee5b-66d0-4b30-af1b-d443d2f91d44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b244ee5b-66d0-4b30-af1b-d443d2f91d44" (UID: "b244ee5b-66d0-4b30-af1b-d443d2f91d44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:05:57.079592 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.079564    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb262e18-01e6-474e-9300-3fa4952ecffa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb262e18-01e6-474e-9300-3fa4952ecffa" (UID: "eb262e18-01e6-474e-9300-3fa4952ecffa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:05:57.079782 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.079757    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc5f5b7b-623d-4754-9ecb-c0a06b13863b" (UID: "bc5f5b7b-623d-4754-9ecb-c0a06b13863b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:05:57.081480 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.081443    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb262e18-01e6-474e-9300-3fa4952ecffa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb262e18-01e6-474e-9300-3fa4952ecffa" (UID: "eb262e18-01e6-474e-9300-3fa4952ecffa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:05:57.081480 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.081469    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b244ee5b-66d0-4b30-af1b-d443d2f91d44-kube-api-access-9wrv5" (OuterVolumeSpecName: "kube-api-access-9wrv5") pod "b244ee5b-66d0-4b30-af1b-d443d2f91d44" (UID: "b244ee5b-66d0-4b30-af1b-d443d2f91d44"). InnerVolumeSpecName "kube-api-access-9wrv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:05:57.081627 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.081478    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-kube-api-access-qjfq7" (OuterVolumeSpecName: "kube-api-access-qjfq7") pod "bc5f5b7b-623d-4754-9ecb-c0a06b13863b" (UID: "bc5f5b7b-623d-4754-9ecb-c0a06b13863b"). InnerVolumeSpecName "kube-api-access-qjfq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:05:57.081627 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.081492    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb262e18-01e6-474e-9300-3fa4952ecffa-kube-api-access-jb2mf" (OuterVolumeSpecName: "kube-api-access-jb2mf") pod "eb262e18-01e6-474e-9300-3fa4952ecffa" (UID: "eb262e18-01e6-474e-9300-3fa4952ecffa"). InnerVolumeSpecName "kube-api-access-jb2mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:05:57.180210 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.180100    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjfq7\" (UniqueName: \"kubernetes.io/projected/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-kube-api-access-qjfq7\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:05:57.180210 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.180185    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b244ee5b-66d0-4b30-af1b-d443d2f91d44-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:05:57.180210 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.180196    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb262e18-01e6-474e-9300-3fa4952ecffa-openshift-service-ca\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:05:57.180210 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.180204    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb2mf\" (UniqueName: \"kubernetes.io/projected/eb262e18-01e6-474e-9300-3fa4952ecffa-kube-api-access-jb2mf\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:05:57.180210 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.180213    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wrv5\" (UniqueName: \"kubernetes.io/projected/b244ee5b-66d0-4b30-af1b-d443d2f91d44-kube-api-access-9wrv5\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:05:57.180496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.180221    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc5f5b7b-623d-4754-9ecb-c0a06b13863b-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:05:57.180496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.180231    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb262e18-01e6-474e-9300-3fa4952ecffa-maas-api-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:05:57.574532 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.574492    2561 generic.go:356] "Generic (PLEG): container finished" podID="b244ee5b-66d0-4b30-af1b-d443d2f91d44" containerID="52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb" exitCode=0
Jul 31 22:05:57.574532 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.574537    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"
Jul 31 22:05:57.574831 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.574573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v" event={"ID":"b244ee5b-66d0-4b30-af1b-d443d2f91d44","Type":"ContainerDied","Data":"52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb"}
Jul 31 22:05:57.574831 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.574630    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v" event={"ID":"b244ee5b-66d0-4b30-af1b-d443d2f91d44","Type":"ContainerDied","Data":"d8007242aaa48ba21a6f76301c4860982317c7d779cde5001cd2016bc6297b38"}
Jul 31 22:05:57.574831 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.574661    2561 scope.go:117] "RemoveContainer" containerID="52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb"
Jul 31 22:05:57.576060 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.576040    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc5f5b7b-623d-4754-9ecb-c0a06b13863b" containerID="4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3" exitCode=0
Jul 31 22:05:57.576231 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.576101    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"
Jul 31 22:05:57.576231 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.576119    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt" event={"ID":"bc5f5b7b-623d-4754-9ecb-c0a06b13863b","Type":"ContainerDied","Data":"4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3"}
Jul 31 22:05:57.576231 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.576185    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt" event={"ID":"bc5f5b7b-623d-4754-9ecb-c0a06b13863b","Type":"ContainerDied","Data":"a5a37a937842139030b8163d7a24e34231dbc4e7edd1253adefcceefe76feab1"}
Jul 31 22:05:57.577709 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.577678    2561 generic.go:356] "Generic (PLEG): container finished" podID="eb262e18-01e6-474e-9300-3fa4952ecffa" containerID="c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49" exitCode=0
Jul 31 22:05:57.577828 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.577735    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"
Jul 31 22:05:57.577828 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.577755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt" event={"ID":"eb262e18-01e6-474e-9300-3fa4952ecffa","Type":"ContainerDied","Data":"c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49"}
Jul 31 22:05:57.577828 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.577804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt" event={"ID":"eb262e18-01e6-474e-9300-3fa4952ecffa","Type":"ContainerDied","Data":"3931265091b65c933444ec1cb6d32a7d6fb58c4952a5d0ba2e45e4a8fea31bc3"}
Jul 31 22:05:57.584359 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.584333    2561 scope.go:117] "RemoveContainer" containerID="52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb"
Jul 31 22:05:57.584729 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:05:57.584707    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb\": container with ID starting with 52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb not found: ID does not exist" containerID="52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb"
Jul 31 22:05:57.584811 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.584737    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb"} err="failed to get container status \"52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb\": rpc error: code = NotFound desc = could not find container \"52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb\": container with ID starting with 52853301448e03c441551ec03079ffb2ad902ea08b88c857dc1e576097596fcb not found: ID does not exist"
Jul 31 22:05:57.584811 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.584762    2561 scope.go:117] "RemoveContainer" containerID="4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3"
Jul 31 22:05:57.594167 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.594020    2561 scope.go:117] "RemoveContainer" containerID="4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3"
Jul 31 22:05:57.594421 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:05:57.594326    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3\": container with ID starting with 4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3 not found: ID does not exist" containerID="4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3"
Jul 31 22:05:57.594421 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.594348    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3"} err="failed to get container status \"4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3\": rpc error: code = NotFound desc = could not find container \"4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3\": container with ID starting with 4ec173e860b0135aa3b3fe6a01b51289e3949d49d1bd8f36025283eb3444ddf3 not found: ID does not exist"
Jul 31 22:05:57.594421 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.594366    2561 scope.go:117] "RemoveContainer" containerID="c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49"
Jul 31 22:05:57.601879 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.601854    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"]
Jul 31 22:05:57.609883 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.609835    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef199660-578b77f648-dlktt"]
Jul 31 22:05:57.616753 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.616731    2561 scope.go:117] "RemoveContainer" containerID="c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49"
Jul 31 22:05:57.617125 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:05:57.617103    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49\": container with ID starting with c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49 not found: ID does not exist" containerID="c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49"
Jul 31 22:05:57.617202 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.617154    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49"} err="failed to get container status \"c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49\": rpc error: code = NotFound desc = could not find container \"c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49\": container with ID starting with c000bd6bc36c2e8649f6bc4b0e24403bc533758606a83650096e1177485d4d49 not found: ID does not exist"
Jul 31 22:05:57.621253 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.621230    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"]
Jul 31 22:05:57.625598 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.625579    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef199660-577f9f5674-dpmgt"]
Jul 31 22:05:57.637410 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.637385    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"]
Jul 31 22:05:57.639863 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.639836    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef199660-5cdf874648-hsf9v"]
Jul 31 22:05:57.927015 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.926927    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"]
Jul 31 22:05:57.927357 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.927343    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5f5b7b-623d-4754-9ecb-c0a06b13863b" containerName="payload-processing"
Jul 31 22:05:57.927409 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.927360    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5f5b7b-623d-4754-9ecb-c0a06b13863b" containerName="payload-processing"
Jul 31 22:05:57.927409 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.927375    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b244ee5b-66d0-4b30-af1b-d443d2f91d44" containerName="payload-pre-processing"
Jul 31 22:05:57.927409 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.927381    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b244ee5b-66d0-4b30-af1b-d443d2f91d44" containerName="payload-pre-processing"
Jul 31 22:05:57.927409 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.927390    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb262e18-01e6-474e-9300-3fa4952ecffa" containerName="maas-api"
Jul 31 22:05:57.927409 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.927396    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb262e18-01e6-474e-9300-3fa4952ecffa" containerName="maas-api"
Jul 31 22:05:57.927571 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.927472    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb262e18-01e6-474e-9300-3fa4952ecffa" containerName="maas-api"
Jul 31 22:05:57.927571 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.927480    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc5f5b7b-623d-4754-9ecb-c0a06b13863b" containerName="payload-processing"
Jul 31 22:05:57.927571 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.927488    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b244ee5b-66d0-4b30-af1b-d443d2f91d44" containerName="payload-pre-processing"
Jul 31 22:05:57.931684 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.931667    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:57.933689 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.933668    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 22:05:57.933811 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.933725    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4544f640-gw-openshift-default-dockercfg-695qs\""
Jul 31 22:05:57.944986 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:57.944958    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"]
Jul 31 22:05:58.088881 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.088846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-envoy\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.089091 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.088904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w6ct\" (UniqueName: \"kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-kube-api-access-6w6ct\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.089091 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.088963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-data\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.089091 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.089025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-socket\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.089091 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.089052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istiod-ca-cert\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.089359 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.089117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-credential-socket\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.089359 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.089184    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-token\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.089359 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.089216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-certs\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.089359 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.089241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-podinfo\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.189951 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.189855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-credential-socket\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.189951 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.189897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-token\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.189951 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.189919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-certs\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.189951 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.189936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-podinfo\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.190358 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.189987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-envoy\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.190358 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.190013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w6ct\" (UniqueName: \"kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-kube-api-access-6w6ct\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.190358 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.190035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-data\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.190358 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.190085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-socket\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.190358 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.190114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istiod-ca-cert\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.190678 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.190360    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-credential-socket\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.190678 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.190410    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-certs\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.190678 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.190565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-data\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.190678 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.190636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-socket\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.190839 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.190820    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istiod-ca-cert\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.192512 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.192494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-envoy\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.192750 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.192732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-podinfo\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.197728 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.197687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-token\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.197848 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.197761    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w6ct\" (UniqueName: \"kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-kube-api-access-6w6ct\") pod \"e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") " pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.228639 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.228601    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b244ee5b-66d0-4b30-af1b-d443d2f91d44" path="/var/lib/kubelet/pods/b244ee5b-66d0-4b30-af1b-d443d2f91d44/volumes"
Jul 31 22:05:58.229026 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.229012    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc5f5b7b-623d-4754-9ecb-c0a06b13863b" path="/var/lib/kubelet/pods/bc5f5b7b-623d-4754-9ecb-c0a06b13863b/volumes"
Jul 31 22:05:58.229429 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.229414    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb262e18-01e6-474e-9300-3fa4952ecffa" path="/var/lib/kubelet/pods/eb262e18-01e6-474e-9300-3fa4952ecffa/volumes"
Jul 31 22:05:58.242722 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.242694    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:05:58.383503 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.383475    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"]
Jul 31 22:05:58.385671 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:05:58.385643    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf7a8c3_be19_4afe_a841_0cba11d7fa86.slice/crio-333dd46c428251b4224c066ec1d515ee019cd24d830227a393f084af2a83e607 WatchSource:0}: Error finding container 333dd46c428251b4224c066ec1d515ee019cd24d830227a393f084af2a83e607: Status 404 returned error can't find the container with id 333dd46c428251b4224c066ec1d515ee019cd24d830227a393f084af2a83e607
Jul 31 22:05:58.585859 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:05:58.585824    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" event={"ID":"baf7a8c3-be19-4afe-a841-0cba11d7fa86","Type":"ContainerStarted","Data":"333dd46c428251b4224c066ec1d515ee019cd24d830227a393f084af2a83e607"}
Jul 31 22:06:00.927402 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:00.927369    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:06:00.927911 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:00.927890    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:06:00.965074 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:00.965031    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:00.965211 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:00.965110    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:00.965211 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:00.965155    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:01.604433 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:01.604389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" event={"ID":"baf7a8c3-be19-4afe-a841-0cba11d7fa86","Type":"ContainerStarted","Data":"a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb"}
Jul 31 22:06:01.623130 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:01.623071    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" podStartSLOduration=2.045900297 podStartE2EDuration="4.623053134s" totalImagesPullingTime="2.577152837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:06:01.62165574 +0000 UTC m=+1801.884404103" watchObservedRunningTime="2026-07-31 22:06:01.623053134 +0000 UTC m=+1801.885801498"
Jul 31 22:06:02.243217 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:02.243178    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:06:02.248235 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:02.248202    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:06:02.614851 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:02.611766    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:06:02.616095 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:02.616067    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:06:03.181289 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.181254    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"]
Jul 31 22:06:03.184707 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.184683    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:03.187497 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.187475    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4544f640-serving-cert\""
Jul 31 22:06:03.196792 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.196765    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"]
Jul 31 22:06:03.338510 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.338467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls\") pod \"maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:03.338510 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.338510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjmzp\" (UniqueName: \"kubernetes.io/projected/c9af52a0-728c-4ae5-b341-2db2b176d033-kube-api-access-cjmzp\") pod \"maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:03.339030 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.338694    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9af52a0-728c-4ae5-b341-2db2b176d033-openshift-service-ca\") pod \"maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:03.439562 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.439475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9af52a0-728c-4ae5-b341-2db2b176d033-openshift-service-ca\") pod \"maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:03.439562 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.439528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls\") pod \"maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:03.439562 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.439546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjmzp\" (UniqueName: \"kubernetes.io/projected/c9af52a0-728c-4ae5-b341-2db2b176d033-kube-api-access-cjmzp\") pod \"maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:03.439780 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:06:03.439641    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-serving-cert: secret "maas-api-e2e-migrate-4544f640-serving-cert" not found
Jul 31 22:06:03.439780 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:06:03.439736    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls podName:c9af52a0-728c-4ae5-b341-2db2b176d033 nodeName:}" failed. No retries permitted until 2026-07-31 22:06:03.939716251 +0000 UTC m=+1804.202464609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls") pod "maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b" (UID: "c9af52a0-728c-4ae5-b341-2db2b176d033") : secret "maas-api-e2e-migrate-4544f640-serving-cert" not found
Jul 31 22:06:03.440106 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.440087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9af52a0-728c-4ae5-b341-2db2b176d033-openshift-service-ca\") pod \"maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:03.451789 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.451753    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjmzp\" (UniqueName: \"kubernetes.io/projected/c9af52a0-728c-4ae5-b341-2db2b176d033-kube-api-access-cjmzp\") pod \"maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:03.870225 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.870190    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"]
Jul 31 22:06:03.873781 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.873764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:03.876277 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.876250    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4544f640\""
Jul 31 22:06:03.876397 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.876301    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4544f640-dockercfg-8kvpl\""
Jul 31 22:06:03.885560 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.885531    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"]
Jul 31 22:06:03.887377 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.887353    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"]
Jul 31 22:06:03.891063 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.891041    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:03.899866 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.899832    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"]
Jul 31 22:06:03.944218 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:03.944180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls\") pod \"maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:03.944397 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:06:03.944334    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-serving-cert: secret "maas-api-e2e-migrate-4544f640-serving-cert" not found
Jul 31 22:06:03.944448 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:06:03.944408    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls podName:c9af52a0-728c-4ae5-b341-2db2b176d033 nodeName:}" failed. No retries permitted until 2026-07-31 22:06:04.944391907 +0000 UTC m=+1805.207140249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls") pod "maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b" (UID: "c9af52a0-728c-4ae5-b341-2db2b176d033") : secret "maas-api-e2e-migrate-4544f640-serving-cert" not found
Jul 31 22:06:04.045759 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.045718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvcx\" (UniqueName: \"kubernetes.io/projected/48fb8586-6a6b-4b11-9771-1476a404e6d4-kube-api-access-vlvcx\") pod \"payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww\" (UID: \"48fb8586-6a6b-4b11-9771-1476a404e6d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:04.045946 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.045859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fb8586-6a6b-4b11-9771-1476a404e6d4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww\" (UID: \"48fb8586-6a6b-4b11-9771-1476a404e6d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:04.045946 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.045921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3449a63-83fa-48b8-878a-5656fe022ff1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct\" (UID: \"a3449a63-83fa-48b8-878a-5656fe022ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:04.046061 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.046045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27bh\" (UniqueName: \"kubernetes.io/projected/a3449a63-83fa-48b8-878a-5656fe022ff1-kube-api-access-x27bh\") pod \"payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct\" (UID: \"a3449a63-83fa-48b8-878a-5656fe022ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:04.146614 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.146514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3449a63-83fa-48b8-878a-5656fe022ff1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct\" (UID: \"a3449a63-83fa-48b8-878a-5656fe022ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:04.146614 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.146597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27bh\" (UniqueName: \"kubernetes.io/projected/a3449a63-83fa-48b8-878a-5656fe022ff1-kube-api-access-x27bh\") pod \"payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct\" (UID: \"a3449a63-83fa-48b8-878a-5656fe022ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:04.146614 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.146619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvcx\" (UniqueName: \"kubernetes.io/projected/48fb8586-6a6b-4b11-9771-1476a404e6d4-kube-api-access-vlvcx\") pod \"payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww\" (UID: \"48fb8586-6a6b-4b11-9771-1476a404e6d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:04.146913 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.146677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fb8586-6a6b-4b11-9771-1476a404e6d4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww\" (UID: \"48fb8586-6a6b-4b11-9771-1476a404e6d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:04.147315 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.147293    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3449a63-83fa-48b8-878a-5656fe022ff1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct\" (UID: \"a3449a63-83fa-48b8-878a-5656fe022ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:04.147372 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.147335    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fb8586-6a6b-4b11-9771-1476a404e6d4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww\" (UID: \"48fb8586-6a6b-4b11-9771-1476a404e6d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:04.155407 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.155378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27bh\" (UniqueName: \"kubernetes.io/projected/a3449a63-83fa-48b8-878a-5656fe022ff1-kube-api-access-x27bh\") pod \"payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct\" (UID: \"a3449a63-83fa-48b8-878a-5656fe022ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:04.155539 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.155390    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvcx\" (UniqueName: \"kubernetes.io/projected/48fb8586-6a6b-4b11-9771-1476a404e6d4-kube-api-access-vlvcx\") pod \"payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww\" (UID: \"48fb8586-6a6b-4b11-9771-1476a404e6d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:04.186673 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.186635    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:04.203340 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.203297    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:04.349482 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.349434    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"]
Jul 31 22:06:04.351818 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:06:04.351785    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3449a63_83fa_48b8_878a_5656fe022ff1.slice/crio-47e9448eec688f6468a2da13bcfad296b4604988076221dbf78a1b750ee4ad14 WatchSource:0}: Error finding container 47e9448eec688f6468a2da13bcfad296b4604988076221dbf78a1b750ee4ad14: Status 404 returned error can't find the container with id 47e9448eec688f6468a2da13bcfad296b4604988076221dbf78a1b750ee4ad14
Jul 31 22:06:04.378329 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.378302    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"]
Jul 31 22:06:04.381464 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:06:04.381417    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fb8586_6a6b_4b11_9771_1476a404e6d4.slice/crio-bf7fcd7b0dc3ae5d8532e68ae7a721a6e8bbec21eeafc8963a6a1d4236184ddf WatchSource:0}: Error finding container bf7fcd7b0dc3ae5d8532e68ae7a721a6e8bbec21eeafc8963a6a1d4236184ddf: Status 404 returned error can't find the container with id bf7fcd7b0dc3ae5d8532e68ae7a721a6e8bbec21eeafc8963a6a1d4236184ddf
Jul 31 22:06:04.622382 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.622330    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct" event={"ID":"a3449a63-83fa-48b8-878a-5656fe022ff1","Type":"ContainerStarted","Data":"9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36"}
Jul 31 22:06:04.622579 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.622405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct" event={"ID":"a3449a63-83fa-48b8-878a-5656fe022ff1","Type":"ContainerStarted","Data":"47e9448eec688f6468a2da13bcfad296b4604988076221dbf78a1b750ee4ad14"}
Jul 31 22:06:04.622579 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.622437    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:04.624035 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.624002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww" event={"ID":"48fb8586-6a6b-4b11-9771-1476a404e6d4","Type":"ContainerStarted","Data":"3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add"}
Jul 31 22:06:04.624225 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.624057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww" event={"ID":"48fb8586-6a6b-4b11-9771-1476a404e6d4","Type":"ContainerStarted","Data":"bf7fcd7b0dc3ae5d8532e68ae7a721a6e8bbec21eeafc8963a6a1d4236184ddf"}
Jul 31 22:06:04.624225 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.624178    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:04.640236 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.640182    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct" podStartSLOduration=1.640164988 podStartE2EDuration="1.640164988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:06:04.638814091 +0000 UTC m=+1804.901562464" watchObservedRunningTime="2026-07-31 22:06:04.640164988 +0000 UTC m=+1804.902913352"
Jul 31 22:06:04.654838 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.654733    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww" podStartSLOduration=1.6547161510000001 podStartE2EDuration="1.654716151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:06:04.653103873 +0000 UTC m=+1804.915852236" watchObservedRunningTime="2026-07-31 22:06:04.654716151 +0000 UTC m=+1804.917464515"
Jul 31 22:06:04.956446 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.956342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls\") pod \"maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:04.958833 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.958806    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls\") pod \"maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:04.995884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:04.995842    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:05.143549 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:05.143495    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"]
Jul 31 22:06:05.145982 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:06:05.145951    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9af52a0_728c_4ae5_b341_2db2b176d033.slice/crio-5d825657ca6ea3f3e8e50bc71241e35542766877cd56f942692a58ac27faeb78 WatchSource:0}: Error finding container 5d825657ca6ea3f3e8e50bc71241e35542766877cd56f942692a58ac27faeb78: Status 404 returned error can't find the container with id 5d825657ca6ea3f3e8e50bc71241e35542766877cd56f942692a58ac27faeb78
Jul 31 22:06:05.631842 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:05.631803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b" event={"ID":"c9af52a0-728c-4ae5-b341-2db2b176d033","Type":"ContainerStarted","Data":"5d825657ca6ea3f3e8e50bc71241e35542766877cd56f942692a58ac27faeb78"}
Jul 31 22:06:06.638359 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:06.638311    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b" event={"ID":"c9af52a0-728c-4ae5-b341-2db2b176d033","Type":"ContainerStarted","Data":"694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31"}
Jul 31 22:06:06.638806 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:06.638472    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:06.655117 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:06.655059    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b" podStartSLOduration=3.236918887 podStartE2EDuration="3.655039955s" totalImagesPullingTime="418.121068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:06:06.653374186 +0000 UTC m=+1806.916122549" watchObservedRunningTime="2026-07-31 22:06:06.655039955 +0000 UTC m=+1806.917788318"
Jul 31 22:06:12.650567 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:12.650536    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:14.679561 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:14.679525    2561 generic.go:356] "Generic (PLEG): container finished" podID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerID="a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb" exitCode=0
Jul 31 22:06:14.679977 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:14.679602    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" event={"ID":"baf7a8c3-be19-4afe-a841-0cba11d7fa86","Type":"ContainerDied","Data":"a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb"}
Jul 31 22:06:14.680070 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:14.680051    2561 scope.go:117] "RemoveContainer" containerID="a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb"
Jul 31 22:06:14.681092 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:14.681061    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:14.681188 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:14.681167    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:14.681234 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:14.681202    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:15.634180 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:15.634123    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:15.634335 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:15.634232    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:15.697158 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:15.688933    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" event={"ID":"baf7a8c3-be19-4afe-a841-0cba11d7fa86","Type":"ContainerStarted","Data":"863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817"}
Jul 31 22:06:16.243428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:16.243391    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:06:16.244640 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:16.244609    2561 patch_prober.go:28] interesting pod/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Jul 31 22:06:16.244769 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:16.244659    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 31 22:06:17.243402 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:17.243363    2561 patch_prober.go:28] interesting pod/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Jul 31 22:06:17.243798 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:17.243420    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 31 22:06:17.611814 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:17.611762    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:06:18.243435 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:18.243400    2561 patch_prober.go:28] interesting pod/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Jul 31 22:06:18.243837 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:18.243456    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 31 22:06:19.243991 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:19.243955    2561 patch_prober.go:28] interesting pod/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Jul 31 22:06:19.244430 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:19.244007    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 31 22:06:20.243734 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:20.243698    2561 patch_prober.go:28] interesting pod/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Jul 31 22:06:20.243909 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:20.243757    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 31 22:06:21.247071 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:21.247040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:06:21.247938 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:21.247917    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:06:37.991245 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:37.991210    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"]
Jul 31 22:06:37.991778 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:37.991475    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b" podUID="c9af52a0-728c-4ae5-b341-2db2b176d033" containerName="maas-api" containerID="cri-o://694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31" gracePeriod=30
Jul 31 22:06:38.084746 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.084718    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"]
Jul 31 22:06:38.084993 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.084953    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww" podUID="48fb8586-6a6b-4b11-9771-1476a404e6d4" containerName="payload-processing" containerID="cri-o://3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add" gracePeriod=30
Jul 31 22:06:38.090452 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.090425    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"]
Jul 31 22:06:38.090664 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.090640    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct" podUID="a3449a63-83fa-48b8-878a-5656fe022ff1" containerName="payload-pre-processing" containerID="cri-o://9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36" gracePeriod=30
Jul 31 22:06:38.251270 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.251214    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:38.361329 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.361303    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:38.377005 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.376981    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:38.379794 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.379771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls\") pod \"c9af52a0-728c-4ae5-b341-2db2b176d033\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") "
Jul 31 22:06:38.379924 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.379872    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjmzp\" (UniqueName: \"kubernetes.io/projected/c9af52a0-728c-4ae5-b341-2db2b176d033-kube-api-access-cjmzp\") pod \"c9af52a0-728c-4ae5-b341-2db2b176d033\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") "
Jul 31 22:06:38.379924 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.379892    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9af52a0-728c-4ae5-b341-2db2b176d033-openshift-service-ca\") pod \"c9af52a0-728c-4ae5-b341-2db2b176d033\" (UID: \"c9af52a0-728c-4ae5-b341-2db2b176d033\") "
Jul 31 22:06:38.380291 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.380267    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9af52a0-728c-4ae5-b341-2db2b176d033-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9af52a0-728c-4ae5-b341-2db2b176d033" (UID: "c9af52a0-728c-4ae5-b341-2db2b176d033"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:06:38.382457 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.382425    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9af52a0-728c-4ae5-b341-2db2b176d033-kube-api-access-cjmzp" (OuterVolumeSpecName: "kube-api-access-cjmzp") pod "c9af52a0-728c-4ae5-b341-2db2b176d033" (UID: "c9af52a0-728c-4ae5-b341-2db2b176d033"). InnerVolumeSpecName "kube-api-access-cjmzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:06:38.382573 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.382451    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9af52a0-728c-4ae5-b341-2db2b176d033" (UID: "c9af52a0-728c-4ae5-b341-2db2b176d033"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:06:38.446818 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.446779    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"]
Jul 31 22:06:38.447301 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.447047    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy" containerID="cri-o://863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817" gracePeriod=30
Jul 31 22:06:38.481163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.480695    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fb8586-6a6b-4b11-9771-1476a404e6d4-plugins-config-volume\") pod \"48fb8586-6a6b-4b11-9771-1476a404e6d4\" (UID: \"48fb8586-6a6b-4b11-9771-1476a404e6d4\") "
Jul 31 22:06:38.481163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.480739    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlvcx\" (UniqueName: \"kubernetes.io/projected/48fb8586-6a6b-4b11-9771-1476a404e6d4-kube-api-access-vlvcx\") pod \"48fb8586-6a6b-4b11-9771-1476a404e6d4\" (UID: \"48fb8586-6a6b-4b11-9771-1476a404e6d4\") "
Jul 31 22:06:38.481163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.480776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x27bh\" (UniqueName: \"kubernetes.io/projected/a3449a63-83fa-48b8-878a-5656fe022ff1-kube-api-access-x27bh\") pod \"a3449a63-83fa-48b8-878a-5656fe022ff1\" (UID: \"a3449a63-83fa-48b8-878a-5656fe022ff1\") "
Jul 31 22:06:38.481163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.480835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3449a63-83fa-48b8-878a-5656fe022ff1-plugins-config-volume\") pod \"a3449a63-83fa-48b8-878a-5656fe022ff1\" (UID: \"a3449a63-83fa-48b8-878a-5656fe022ff1\") "
Jul 31 22:06:38.481163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.481107    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjmzp\" (UniqueName: \"kubernetes.io/projected/c9af52a0-728c-4ae5-b341-2db2b176d033-kube-api-access-cjmzp\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:38.481163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.481125    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9af52a0-728c-4ae5-b341-2db2b176d033-openshift-service-ca\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:38.481163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.481154    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9af52a0-728c-4ae5-b341-2db2b176d033-maas-api-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:38.481706 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.481537    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3449a63-83fa-48b8-878a-5656fe022ff1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3449a63-83fa-48b8-878a-5656fe022ff1" (UID: "a3449a63-83fa-48b8-878a-5656fe022ff1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:06:38.481935 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.481909    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fb8586-6a6b-4b11-9771-1476a404e6d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "48fb8586-6a6b-4b11-9771-1476a404e6d4" (UID: "48fb8586-6a6b-4b11-9771-1476a404e6d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:06:38.483858 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.483834    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fb8586-6a6b-4b11-9771-1476a404e6d4-kube-api-access-vlvcx" (OuterVolumeSpecName: "kube-api-access-vlvcx") pod "48fb8586-6a6b-4b11-9771-1476a404e6d4" (UID: "48fb8586-6a6b-4b11-9771-1476a404e6d4"). InnerVolumeSpecName "kube-api-access-vlvcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:06:38.486000 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.485967    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3449a63-83fa-48b8-878a-5656fe022ff1-kube-api-access-x27bh" (OuterVolumeSpecName: "kube-api-access-x27bh") pod "a3449a63-83fa-48b8-878a-5656fe022ff1" (UID: "a3449a63-83fa-48b8-878a-5656fe022ff1"). InnerVolumeSpecName "kube-api-access-x27bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:06:38.582380 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.582351    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fb8586-6a6b-4b11-9771-1476a404e6d4-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:38.582380 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.582377    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvcx\" (UniqueName: \"kubernetes.io/projected/48fb8586-6a6b-4b11-9771-1476a404e6d4-kube-api-access-vlvcx\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:38.582561 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.582387    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x27bh\" (UniqueName: \"kubernetes.io/projected/a3449a63-83fa-48b8-878a-5656fe022ff1-kube-api-access-x27bh\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:38.582561 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.582396    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3449a63-83fa-48b8-878a-5656fe022ff1-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:38.815378 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.815341    2561 generic.go:356] "Generic (PLEG): container finished" podID="a3449a63-83fa-48b8-878a-5656fe022ff1" containerID="9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36" exitCode=0
Jul 31 22:06:38.815577 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.815412    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"
Jul 31 22:06:38.815577 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.815428    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct" event={"ID":"a3449a63-83fa-48b8-878a-5656fe022ff1","Type":"ContainerDied","Data":"9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36"}
Jul 31 22:06:38.815577 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.815518    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct" event={"ID":"a3449a63-83fa-48b8-878a-5656fe022ff1","Type":"ContainerDied","Data":"47e9448eec688f6468a2da13bcfad296b4604988076221dbf78a1b750ee4ad14"}
Jul 31 22:06:38.815577 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.815549    2561 scope.go:117] "RemoveContainer" containerID="9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36"
Jul 31 22:06:38.816900 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.816872    2561 generic.go:356] "Generic (PLEG): container finished" podID="48fb8586-6a6b-4b11-9771-1476a404e6d4" containerID="3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add" exitCode=0
Jul 31 22:06:38.817032 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.816927    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"
Jul 31 22:06:38.817032 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.816954    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww" event={"ID":"48fb8586-6a6b-4b11-9771-1476a404e6d4","Type":"ContainerDied","Data":"3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add"}
Jul 31 22:06:38.817170 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.817045    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww" event={"ID":"48fb8586-6a6b-4b11-9771-1476a404e6d4","Type":"ContainerDied","Data":"bf7fcd7b0dc3ae5d8532e68ae7a721a6e8bbec21eeafc8963a6a1d4236184ddf"}
Jul 31 22:06:38.818609 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.818584    2561 generic.go:356] "Generic (PLEG): container finished" podID="c9af52a0-728c-4ae5-b341-2db2b176d033" containerID="694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31" exitCode=0
Jul 31 22:06:38.818785 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.818626    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b" event={"ID":"c9af52a0-728c-4ae5-b341-2db2b176d033","Type":"ContainerDied","Data":"694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31"}
Jul 31 22:06:38.818892 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.818814    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b" event={"ID":"c9af52a0-728c-4ae5-b341-2db2b176d033","Type":"ContainerDied","Data":"5d825657ca6ea3f3e8e50bc71241e35542766877cd56f942692a58ac27faeb78"}
Jul 31 22:06:38.818892 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.818651    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"
Jul 31 22:06:38.827071 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.827054    2561 scope.go:117] "RemoveContainer" containerID="9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36"
Jul 31 22:06:38.827438 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:06:38.827415    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36\": container with ID starting with 9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36 not found: ID does not exist" containerID="9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36"
Jul 31 22:06:38.827902 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.827710    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36"} err="failed to get container status \"9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36\": rpc error: code = NotFound desc = could not find container \"9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36\": container with ID starting with 9e5591a4a16e9b5efe6ad3e3578618a603937e7c92271c1f964eb5845fba5a36 not found: ID does not exist"
Jul 31 22:06:38.827902 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.827738    2561 scope.go:117] "RemoveContainer" containerID="3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add"
Jul 31 22:06:38.836052 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.836034    2561 scope.go:117] "RemoveContainer" containerID="3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add"
Jul 31 22:06:38.836343 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:06:38.836324    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add\": container with ID starting with 3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add not found: ID does not exist" containerID="3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add"
Jul 31 22:06:38.836389 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.836353    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add"} err="failed to get container status \"3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add\": rpc error: code = NotFound desc = could not find container \"3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add\": container with ID starting with 3e572c116d2a4e5d7ef81d76b76622f4ad92a5dc8265df2ce84f43d4f6066add not found: ID does not exist"
Jul 31 22:06:38.836389 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.836373    2561 scope.go:117] "RemoveContainer" containerID="694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31"
Jul 31 22:06:38.843392 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.843368    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"]
Jul 31 22:06:38.844830 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.844812    2561 scope.go:117] "RemoveContainer" containerID="694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31"
Jul 31 22:06:38.845111 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:06:38.845095    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31\": container with ID starting with 694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31 not found: ID does not exist" containerID="694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31"
Jul 31 22:06:38.845172 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.845116    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31"} err="failed to get container status \"694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31\": rpc error: code = NotFound desc = could not find container \"694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31\": container with ID starting with 694e9055a0af6ff6ba7ff8aab8f51eeccbfad2a12793498f2325e5658fbd3f31 not found: ID does not exist"
Jul 31 22:06:38.848408 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.848387    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4544f640-76cd6b5996-fw4ct"]
Jul 31 22:06:38.858292 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.858272    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"]
Jul 31 22:06:38.861952 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.861922    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4544f640-67ccd5cf4f-cbrww"]
Jul 31 22:06:38.871921 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.871897    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"]
Jul 31 22:06:38.873522 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:38.873503    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4544f640-6d966dbf94-rd69b"]
Jul 31 22:06:39.194509 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.194425    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"]
Jul 31 22:06:39.194927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.194803    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48fb8586-6a6b-4b11-9771-1476a404e6d4" containerName="payload-processing"
Jul 31 22:06:39.194927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.194814    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="48fb8586-6a6b-4b11-9771-1476a404e6d4" containerName="payload-processing"
Jul 31 22:06:39.194927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.194836    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9af52a0-728c-4ae5-b341-2db2b176d033" containerName="maas-api"
Jul 31 22:06:39.194927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.194842    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9af52a0-728c-4ae5-b341-2db2b176d033" containerName="maas-api"
Jul 31 22:06:39.194927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.194849    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3449a63-83fa-48b8-878a-5656fe022ff1" containerName="payload-pre-processing"
Jul 31 22:06:39.194927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.194855    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3449a63-83fa-48b8-878a-5656fe022ff1" containerName="payload-pre-processing"
Jul 31 22:06:39.194927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.194923    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9af52a0-728c-4ae5-b341-2db2b176d033" containerName="maas-api"
Jul 31 22:06:39.194927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.194932    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="48fb8586-6a6b-4b11-9771-1476a404e6d4" containerName="payload-processing"
Jul 31 22:06:39.195383 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.194938    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3449a63-83fa-48b8-878a-5656fe022ff1" containerName="payload-pre-processing"
Jul 31 22:06:39.199335 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.199313    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.202243 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.202222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8daf3c2f-openshift-default-dockercfg-q2hc5\""
Jul 31 22:06:39.208269 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.208246    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"]
Jul 31 22:06:39.287543 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.287509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-socket\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.287697 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.287561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-token\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.287765 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.287700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-envoy\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.287842 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.287823    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-certs\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.287890 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.287868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-data\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.287939 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.287897    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47xzt\" (UniqueName: \"kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-kube-api-access-47xzt\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.287986 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.287950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-credential-socket\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.288026 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.288009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-podinfo\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.288087 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.288073    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istiod-ca-cert\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.388860 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.388831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-socket\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389049 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.388869    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-token\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389049 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.388906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-envoy\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389049 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.388931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-certs\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389049 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.388963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-data\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389049 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.388989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47xzt\" (UniqueName: \"kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-kube-api-access-47xzt\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389049 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.389017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-credential-socket\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389398 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.389057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-podinfo\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389398 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.389100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istiod-ca-cert\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389398 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.389335    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-socket\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389398 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.389361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-certs\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389573 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.389451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-data\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389573 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.389536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-credential-socket\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.389857 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.389839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istiod-ca-cert\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.391262 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.391241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-envoy\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.391553 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.391535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-podinfo\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.396182 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.396158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-token\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.396293 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.396279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47xzt\" (UniqueName: \"kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-kube-api-access-47xzt\") pod \"e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") " pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.510641 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.510558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:39.645038 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.645014    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"]
Jul 31 22:06:39.647175 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:06:39.647148    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf405d4_45d0_46d4_a580_34b0c50f83fd.slice/crio-6a7bd95c282bab256abbb04770ea0e305b9eebce0495743336c23f339714bd40 WatchSource:0}: Error finding container 6a7bd95c282bab256abbb04770ea0e305b9eebce0495743336c23f339714bd40: Status 404 returned error can't find the container with id 6a7bd95c282bab256abbb04770ea0e305b9eebce0495743336c23f339714bd40
Jul 31 22:06:39.649276 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.649245    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:39.649389 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.649325    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:39.649389 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.649373    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:39.831677 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.830766    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" event={"ID":"aaf405d4-45d0-46d4-a580-34b0c50f83fd","Type":"ContainerStarted","Data":"e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79"}
Jul 31 22:06:39.831677 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.830831    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" event={"ID":"aaf405d4-45d0-46d4-a580-34b0c50f83fd","Type":"ContainerStarted","Data":"6a7bd95c282bab256abbb04770ea0e305b9eebce0495743336c23f339714bd40"}
Jul 31 22:06:39.851605 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:39.851536    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" podStartSLOduration=0.85151508 podStartE2EDuration="851.51508ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:06:39.848981749 +0000 UTC m=+1840.111730114" watchObservedRunningTime="2026-07-31 22:06:39.85151508 +0000 UTC m=+1840.114263445"
Jul 31 22:06:40.228540 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:40.228463    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48fb8586-6a6b-4b11-9771-1476a404e6d4" path="/var/lib/kubelet/pods/48fb8586-6a6b-4b11-9771-1476a404e6d4/volumes"
Jul 31 22:06:40.228883 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:40.228839    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3449a63-83fa-48b8-878a-5656fe022ff1" path="/var/lib/kubelet/pods/a3449a63-83fa-48b8-878a-5656fe022ff1/volumes"
Jul 31 22:06:40.229200 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:40.229187    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9af52a0-728c-4ae5-b341-2db2b176d033" path="/var/lib/kubelet/pods/c9af52a0-728c-4ae5-b341-2db2b176d033/volumes"
Jul 31 22:06:40.510808 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:40.510713    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:40.515473 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:40.515449    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:40.837930 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:40.837902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:40.838894 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:40.838874    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:43.044766 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.044730    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"]
Jul 31 22:06:43.049429 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.049411    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:06:43.051406 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.051379    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8daf3c2f-dockercfg-h6xnn\""
Jul 31 22:06:43.051580 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.051564    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8daf3c2f\""
Jul 31 22:06:43.059394 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.059372    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"]
Jul 31 22:06:43.128182 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.128149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/487a920d-95b9-412b-b311-e58ee44e0cd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92\" (UID: \"487a920d-95b9-412b-b311-e58ee44e0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:06:43.128182 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.128181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtjws\" (UniqueName: \"kubernetes.io/projected/487a920d-95b9-412b-b311-e58ee44e0cd9-kube-api-access-qtjws\") pod \"payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92\" (UID: \"487a920d-95b9-412b-b311-e58ee44e0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:06:43.229549 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.229513    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/487a920d-95b9-412b-b311-e58ee44e0cd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92\" (UID: \"487a920d-95b9-412b-b311-e58ee44e0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:06:43.229549 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.229546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjws\" (UniqueName: \"kubernetes.io/projected/487a920d-95b9-412b-b311-e58ee44e0cd9-kube-api-access-qtjws\") pod \"payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92\" (UID: \"487a920d-95b9-412b-b311-e58ee44e0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:06:43.230261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.230242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/487a920d-95b9-412b-b311-e58ee44e0cd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92\" (UID: \"487a920d-95b9-412b-b311-e58ee44e0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:06:43.237612 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.237587    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjws\" (UniqueName: \"kubernetes.io/projected/487a920d-95b9-412b-b311-e58ee44e0cd9-kube-api-access-qtjws\") pod \"payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92\" (UID: \"487a920d-95b9-412b-b311-e58ee44e0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:06:43.360484 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.360404    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:06:43.500410 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.500350    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"]
Jul 31 22:06:43.520991 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:06:43.520962    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487a920d_95b9_412b_b311_e58ee44e0cd9.slice/crio-47244eff9868656c074f5632d6888fc59aee611474ee18132e3fb2722fc19948 WatchSource:0}: Error finding container 47244eff9868656c074f5632d6888fc59aee611474ee18132e3fb2722fc19948: Status 404 returned error can't find the container with id 47244eff9868656c074f5632d6888fc59aee611474ee18132e3fb2722fc19948
Jul 31 22:06:43.679325 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.679294    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:06:43.835326 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835292    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-envoy\") pod \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") "
Jul 31 22:06:43.835513 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835337    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-certs\") pod \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") "
Jul 31 22:06:43.835513 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istiod-ca-cert\") pod \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") "
Jul 31 22:06:43.835513 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835412    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-credential-socket\") pod \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") "
Jul 31 22:06:43.835513 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835433    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w6ct\" (UniqueName: \"kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-kube-api-access-6w6ct\") pod \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") "
Jul 31 22:06:43.835513 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835449    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-token\") pod \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") "
Jul 31 22:06:43.835513 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-podinfo\") pod \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") "
Jul 31 22:06:43.835825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835532    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-data\") pod \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") "
Jul 31 22:06:43.835825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835572    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-socket\") pod \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\" (UID: \"baf7a8c3-be19-4afe-a841-0cba11d7fa86\") "
Jul 31 22:06:43.835825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835668    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "baf7a8c3-be19-4afe-a841-0cba11d7fa86" (UID: "baf7a8c3-be19-4afe-a841-0cba11d7fa86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:06:43.835825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835729    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "baf7a8c3-be19-4afe-a841-0cba11d7fa86" (UID: "baf7a8c3-be19-4afe-a841-0cba11d7fa86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:06:43.836017 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835867    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-data" (OuterVolumeSpecName: "istio-data") pod "baf7a8c3-be19-4afe-a841-0cba11d7fa86" (UID: "baf7a8c3-be19-4afe-a841-0cba11d7fa86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:06:43.836017 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.835933    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "baf7a8c3-be19-4afe-a841-0cba11d7fa86" (UID: "baf7a8c3-be19-4afe-a841-0cba11d7fa86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:06:43.836130 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.836054    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "baf7a8c3-be19-4afe-a841-0cba11d7fa86" (UID: "baf7a8c3-be19-4afe-a841-0cba11d7fa86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:06:43.836508 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.836476    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-data\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:43.836508 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.836499    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:43.836508 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.836512    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-workload-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:43.836689 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.836524    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istiod-ca-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:43.836689 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.836536    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-credential-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:43.837656 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.837633    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "baf7a8c3-be19-4afe-a841-0cba11d7fa86" (UID: "baf7a8c3-be19-4afe-a841-0cba11d7fa86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:06:43.837735 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.837696    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-kube-api-access-6w6ct" (OuterVolumeSpecName: "kube-api-access-6w6ct") pod "baf7a8c3-be19-4afe-a841-0cba11d7fa86" (UID: "baf7a8c3-be19-4afe-a841-0cba11d7fa86"). InnerVolumeSpecName "kube-api-access-6w6ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:06:43.838122 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.838094    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "baf7a8c3-be19-4afe-a841-0cba11d7fa86" (UID: "baf7a8c3-be19-4afe-a841-0cba11d7fa86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 22:06:43.838122 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.838106    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-token" (OuterVolumeSpecName: "istio-token") pod "baf7a8c3-be19-4afe-a841-0cba11d7fa86" (UID: "baf7a8c3-be19-4afe-a841-0cba11d7fa86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:06:43.858257 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.858202    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92" event={"ID":"487a920d-95b9-412b-b311-e58ee44e0cd9","Type":"ContainerStarted","Data":"50efa9157d167aca83a98600337f9e5a5ad87e76dee97557328c2d1c01caf71e"}
Jul 31 22:06:43.858469 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.858449    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:06:43.858550 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.858500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92" event={"ID":"487a920d-95b9-412b-b311-e58ee44e0cd9","Type":"ContainerStarted","Data":"47244eff9868656c074f5632d6888fc59aee611474ee18132e3fb2722fc19948"}
Jul 31 22:06:43.859825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.859804    2561 generic.go:356] "Generic (PLEG): container finished" podID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerID="863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817" exitCode=0
Jul 31 22:06:43.859926 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.859872    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"
Jul 31 22:06:43.859926 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.859886    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" event={"ID":"baf7a8c3-be19-4afe-a841-0cba11d7fa86","Type":"ContainerDied","Data":"863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817"}
Jul 31 22:06:43.859926 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.859925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr" event={"ID":"baf7a8c3-be19-4afe-a841-0cba11d7fa86","Type":"ContainerDied","Data":"333dd46c428251b4224c066ec1d515ee019cd24d830227a393f084af2a83e607"}
Jul 31 22:06:43.860092 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.859946    2561 scope.go:117] "RemoveContainer" containerID="863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817"
Jul 31 22:06:43.868720 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.868702    2561 scope.go:117] "RemoveContainer" containerID="a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb"
Jul 31 22:06:43.876189 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.876166    2561 scope.go:117] "RemoveContainer" containerID="863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817"
Jul 31 22:06:43.876438 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:06:43.876421    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817\": container with ID starting with 863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817 not found: ID does not exist" containerID="863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817"
Jul 31 22:06:43.876491 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.876445    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817"} err="failed to get container status \"863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817\": rpc error: code = NotFound desc = could not find container \"863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817\": container with ID starting with 863b9fb6f2e540ca3432dd572fe6f44af7aa82c10041a773bdec4bf3094a9817 not found: ID does not exist"
Jul 31 22:06:43.876491 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.876461    2561 scope.go:117] "RemoveContainer" containerID="a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb"
Jul 31 22:06:43.876700 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:06:43.876679    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb\": container with ID starting with a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb not found: ID does not exist" containerID="a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb"
Jul 31 22:06:43.876737 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.876708    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb"} err="failed to get container status \"a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb\": rpc error: code = NotFound desc = could not find container \"a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb\": container with ID starting with a9b03fe43b5ac32887a03952b0a35a6fcc299996824b29c92ebece2b7d8f79fb not found: ID does not exist"
Jul 31 22:06:43.880587 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.880549    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92" podStartSLOduration=0.880535901 podStartE2EDuration="880.535901ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:06:43.877735291 +0000 UTC m=+1844.140483654" watchObservedRunningTime="2026-07-31 22:06:43.880535901 +0000 UTC m=+1844.143284265"
Jul 31 22:06:43.893812 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.893788    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"]
Jul 31 22:06:43.899607 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.899582    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4544f640-gw-openshift-default-767dd6df56-lczwr"]
Jul 31 22:06:43.937850 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.937827    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-envoy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:43.937850 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.937851    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w6ct\" (UniqueName: \"kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-kube-api-access-6w6ct\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:43.937992 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.937861    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:43.937992 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:43.937871    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf7a8c3-be19-4afe-a841-0cba11d7fa86-istio-podinfo\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:06:44.227707 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:44.227626    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" path="/var/lib/kubelet/pods/baf7a8c3-be19-4afe-a841-0cba11d7fa86/volumes"
Jul 31 22:06:54.868107 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:54.868079    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:06:54.931826 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:54.931792    2561 generic.go:356] "Generic (PLEG): container finished" podID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerID="e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79" exitCode=0
Jul 31 22:06:54.931986 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:54.931870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" event={"ID":"aaf405d4-45d0-46d4-a580-34b0c50f83fd","Type":"ContainerDied","Data":"e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79"}
Jul 31 22:06:54.932306 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:54.932282    2561 scope.go:117] "RemoveContainer" containerID="e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79"
Jul 31 22:06:54.933398 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:54.933367    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:54.933498 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:54.933441    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:54.933498 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:54.933483    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:06:55.838655 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:55.838621    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:55.938050 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:55.938015    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" event={"ID":"aaf405d4-45d0-46d4-a580-34b0c50f83fd","Type":"ContainerStarted","Data":"9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a"}
Jul 31 22:06:56.511673 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:56.511635    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:06:56.512680 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:56.512656    2561 patch_prober.go:28] interesting pod/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 31 22:06:56.512786 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:56.512712    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 31 22:06:57.511256 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:57.511216    2561 patch_prober.go:28] interesting pod/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 31 22:06:57.511756 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:57.511278    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 31 22:06:58.511960 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:58.511922    2561 patch_prober.go:28] interesting pod/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 31 22:06:58.512451 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:58.511982    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 31 22:06:59.511401 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:59.511362    2561 patch_prober.go:28] interesting pod/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 31 22:06:59.511602 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:06:59.511425    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 31 22:07:00.511076 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:00.511037    2561 patch_prober.go:28] interesting pod/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 31 22:07:00.511559 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:00.511107    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 31 22:07:01.515101 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:01.515065    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:07:01.515523 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:01.515442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:07:01.516079 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:01.516061    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:07:12.888961 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:12.888926    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"]
Jul 31 22:07:12.889566 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:12.889229    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92" podUID="487a920d-95b9-412b-b311-e58ee44e0cd9" containerName="payload-pre-processing" containerID="cri-o://50efa9157d167aca83a98600337f9e5a5ad87e76dee97557328c2d1c01caf71e" gracePeriod=30
Jul 31 22:07:13.023712 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:13.023673    2561 generic.go:356] "Generic (PLEG): container finished" podID="487a920d-95b9-412b-b311-e58ee44e0cd9" containerID="50efa9157d167aca83a98600337f9e5a5ad87e76dee97557328c2d1c01caf71e" exitCode=0
Jul 31 22:07:13.023846 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:13.023709    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92" event={"ID":"487a920d-95b9-412b-b311-e58ee44e0cd9","Type":"ContainerDied","Data":"50efa9157d167aca83a98600337f9e5a5ad87e76dee97557328c2d1c01caf71e"}
Jul 31 22:07:13.151645 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:13.151580    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:07:13.229509 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:13.229476    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/487a920d-95b9-412b-b311-e58ee44e0cd9-plugins-config-volume\") pod \"487a920d-95b9-412b-b311-e58ee44e0cd9\" (UID: \"487a920d-95b9-412b-b311-e58ee44e0cd9\") "
Jul 31 22:07:13.229710 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:13.229519    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtjws\" (UniqueName: \"kubernetes.io/projected/487a920d-95b9-412b-b311-e58ee44e0cd9-kube-api-access-qtjws\") pod \"487a920d-95b9-412b-b311-e58ee44e0cd9\" (UID: \"487a920d-95b9-412b-b311-e58ee44e0cd9\") "
Jul 31 22:07:13.229829 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:13.229807    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487a920d-95b9-412b-b311-e58ee44e0cd9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "487a920d-95b9-412b-b311-e58ee44e0cd9" (UID: "487a920d-95b9-412b-b311-e58ee44e0cd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:07:13.231554 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:13.231529    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487a920d-95b9-412b-b311-e58ee44e0cd9-kube-api-access-qtjws" (OuterVolumeSpecName: "kube-api-access-qtjws") pod "487a920d-95b9-412b-b311-e58ee44e0cd9" (UID: "487a920d-95b9-412b-b311-e58ee44e0cd9"). InnerVolumeSpecName "kube-api-access-qtjws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:07:13.330598 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:13.330550    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/487a920d-95b9-412b-b311-e58ee44e0cd9-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:13.330598 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:13.330592    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtjws\" (UniqueName: \"kubernetes.io/projected/487a920d-95b9-412b-b311-e58ee44e0cd9-kube-api-access-qtjws\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:14.029307 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:14.029274    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"
Jul 31 22:07:14.029307 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:14.029290    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92" event={"ID":"487a920d-95b9-412b-b311-e58ee44e0cd9","Type":"ContainerDied","Data":"47244eff9868656c074f5632d6888fc59aee611474ee18132e3fb2722fc19948"}
Jul 31 22:07:14.029803 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:14.029360    2561 scope.go:117] "RemoveContainer" containerID="50efa9157d167aca83a98600337f9e5a5ad87e76dee97557328c2d1c01caf71e"
Jul 31 22:07:14.050340 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:14.050313    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"]
Jul 31 22:07:14.053925 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:14.053890    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8daf3c2f-674cffffd4-jnc92"]
Jul 31 22:07:14.227955 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:14.227917    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="487a920d-95b9-412b-b311-e58ee44e0cd9" path="/var/lib/kubelet/pods/487a920d-95b9-412b-b311-e58ee44e0cd9/volumes"
Jul 31 22:07:14.254112 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:14.254081    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"]
Jul 31 22:07:14.254319 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:14.254298    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy" containerID="cri-o://9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a" gracePeriod=30
Jul 31 22:07:15.020371 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.020337    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"]
Jul 31 22:07:15.020715 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.020702    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy"
Jul 31 22:07:15.020766 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.020716    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy"
Jul 31 22:07:15.020766 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.020727    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="487a920d-95b9-412b-b311-e58ee44e0cd9" containerName="payload-pre-processing"
Jul 31 22:07:15.020766 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.020733    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="487a920d-95b9-412b-b311-e58ee44e0cd9" containerName="payload-pre-processing"
Jul 31 22:07:15.020766 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.020744    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy"
Jul 31 22:07:15.020766 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.020749    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy"
Jul 31 22:07:15.020922 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.020810    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy"
Jul 31 22:07:15.020922 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.020817    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="487a920d-95b9-412b-b311-e58ee44e0cd9" containerName="payload-pre-processing"
Jul 31 22:07:15.020922 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.020826    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf7a8c3-be19-4afe-a841-0cba11d7fa86" containerName="istio-proxy"
Jul 31 22:07:15.024905 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.024889    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.027094 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.027079    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e3a03c45-openshift-default-dockercfg-l4k7g\""
Jul 31 22:07:15.036991 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.036966    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"]
Jul 31 22:07:15.147632 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.147595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-socket\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.147632 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.147633    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-istio-token\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.147888 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.147654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-credential-socket\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.147888 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.147676    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-data\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.147888 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.147761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08880c5c-b10f-442e-a542-f48922f03f7c-istio-podinfo\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.147888 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.147797    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08880c5c-b10f-442e-a542-f48922f03f7c-istiod-ca-cert\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.148041 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.147887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9l5\" (UniqueName: \"kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-kube-api-access-ft9l5\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.148041 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.147918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-envoy\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.148041 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.147943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-certs\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.248902 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.248872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-socket\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.248902 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.248907    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-istio-token\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249178 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.248929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-credential-socket\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249178 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.248945    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-data\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249178 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.248990    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08880c5c-b10f-442e-a542-f48922f03f7c-istio-podinfo\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249178 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.249014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08880c5c-b10f-442e-a542-f48922f03f7c-istiod-ca-cert\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249388 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.249069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9l5\" (UniqueName: \"kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-kube-api-access-ft9l5\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249449 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.249386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-credential-socket\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249449 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.249398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-socket\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249449 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.249442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-envoy\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249615 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.249495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-certs\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249666 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.249638    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-data\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249800 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.249777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-certs\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.249862 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.249816    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08880c5c-b10f-442e-a542-f48922f03f7c-istiod-ca-cert\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.251472 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.251451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-envoy\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.251716 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.251686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08880c5c-b10f-442e-a542-f48922f03f7c-istio-podinfo\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.258434 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.258408    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-istio-token\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.258894 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.258872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9l5\" (UniqueName: \"kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-kube-api-access-ft9l5\") pod \"e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") " pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.336587 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.336558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:15.469266 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.469232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"]
Jul 31 22:07:15.473337 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:07:15.473307    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08880c5c_b10f_442e_a542_f48922f03f7c.slice/crio-bf55f610827d9170efc9770ae752ce99a1184a3e907bdccff47426cd1e728f27 WatchSource:0}: Error finding container bf55f610827d9170efc9770ae752ce99a1184a3e907bdccff47426cd1e728f27: Status 404 returned error can't find the container with id bf55f610827d9170efc9770ae752ce99a1184a3e907bdccff47426cd1e728f27
Jul 31 22:07:15.475671 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.475641    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:07:15.475749 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.475706    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:07:15.475749 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:15.475736    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:07:16.048983 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:16.048943    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" event={"ID":"08880c5c-b10f-442e-a542-f48922f03f7c","Type":"ContainerStarted","Data":"d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de"}
Jul 31 22:07:16.049485 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:16.049033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" event={"ID":"08880c5c-b10f-442e-a542-f48922f03f7c","Type":"ContainerStarted","Data":"bf55f610827d9170efc9770ae752ce99a1184a3e907bdccff47426cd1e728f27"}
Jul 31 22:07:16.068648 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:16.068602    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" podStartSLOduration=1.068587265 podStartE2EDuration="1.068587265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:07:16.066338096 +0000 UTC m=+1876.329086465" watchObservedRunningTime="2026-07-31 22:07:16.068587265 +0000 UTC m=+1876.331335629"
Jul 31 22:07:16.337247 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:16.337160    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:16.342375 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:16.342346    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:16.518193 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:16.518149    2561 patch_prober.go:28] interesting pod/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 31 22:07:16.518417 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:16.518217    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 22:07:17.054188 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:17.054159    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:17.055321 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:17.055300    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:19.155827 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.155793    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"]
Jul 31 22:07:19.159336 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.159319    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:19.161564 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.161527    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e3a03c45\""
Jul 31 22:07:19.161710 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.161597    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e3a03c45-dockercfg-b4ztw\""
Jul 31 22:07:19.171381 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.171354    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"]
Jul 31 22:07:19.290548 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.290513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt\" (UID: \"60058ec6-fcbd-4625-b93e-4c1d5f43a01a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:19.290766 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.290746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlb96\" (UniqueName: \"kubernetes.io/projected/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-kube-api-access-jlb96\") pod \"payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt\" (UID: \"60058ec6-fcbd-4625-b93e-4c1d5f43a01a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:19.392279 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.392248    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt\" (UID: \"60058ec6-fcbd-4625-b93e-4c1d5f43a01a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:19.392434 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.392300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlb96\" (UniqueName: \"kubernetes.io/projected/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-kube-api-access-jlb96\") pod \"payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt\" (UID: \"60058ec6-fcbd-4625-b93e-4c1d5f43a01a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:19.392941 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.392920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt\" (UID: \"60058ec6-fcbd-4625-b93e-4c1d5f43a01a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:19.400169 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.400127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlb96\" (UniqueName: \"kubernetes.io/projected/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-kube-api-access-jlb96\") pod \"payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt\" (UID: \"60058ec6-fcbd-4625-b93e-4c1d5f43a01a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:19.471267 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.471240    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:19.508096 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.508071    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:07:19.610588 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.610557    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"]
Jul 31 22:07:19.612915 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:07:19.612886    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60058ec6_fcbd_4625_b93e_4c1d5f43a01a.slice/crio-5c0363c6c4e4ff6d0a7ca2398e0a5b6a4b0cb6229ebc79845a6f0d923cdf935f WatchSource:0}: Error finding container 5c0363c6c4e4ff6d0a7ca2398e0a5b6a4b0cb6229ebc79845a6f0d923cdf935f: Status 404 returned error can't find the container with id 5c0363c6c4e4ff6d0a7ca2398e0a5b6a4b0cb6229ebc79845a6f0d923cdf935f
Jul 31 22:07:19.695325 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.695252    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-data\") pod \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") "
Jul 31 22:07:19.695325 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.695289    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-podinfo\") pod \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") "
Jul 31 22:07:19.695516 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.695338    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-envoy\") pod \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") "
Jul 31 22:07:19.695516 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.695386    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-token\") pod \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") "
Jul 31 22:07:19.695516 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.695406    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47xzt\" (UniqueName: \"kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-kube-api-access-47xzt\") pod \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") "
Jul 31 22:07:19.695516 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.695453    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-socket\") pod \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") "
Jul 31 22:07:19.695516 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.695479    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istiod-ca-cert\") pod \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") "
Jul 31 22:07:19.695516 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.695505    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-certs\") pod \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") "
Jul 31 22:07:19.695841 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.695537    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-data" (OuterVolumeSpecName: "istio-data") pod "aaf405d4-45d0-46d4-a580-34b0c50f83fd" (UID: "aaf405d4-45d0-46d4-a580-34b0c50f83fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:07:19.695841 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.695546    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-credential-socket\") pod \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\" (UID: \"aaf405d4-45d0-46d4-a580-34b0c50f83fd\") "
Jul 31 22:07:19.695841 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.695820    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aaf405d4-45d0-46d4-a580-34b0c50f83fd" (UID: "aaf405d4-45d0-46d4-a580-34b0c50f83fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:07:19.696090 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.696065    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aaf405d4-45d0-46d4-a580-34b0c50f83fd" (UID: "aaf405d4-45d0-46d4-a580-34b0c50f83fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:07:19.696249 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.696217    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aaf405d4-45d0-46d4-a580-34b0c50f83fd" (UID: "aaf405d4-45d0-46d4-a580-34b0c50f83fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:07:19.696365 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.696248    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aaf405d4-45d0-46d4-a580-34b0c50f83fd" (UID: "aaf405d4-45d0-46d4-a580-34b0c50f83fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:07:19.696620 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.696598    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-data\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:19.696620 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.696626    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:19.696804 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.696644    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istiod-ca-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:19.696804 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.696658    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-workload-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:19.696804 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.696671    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-credential-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:19.697667 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.697641    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aaf405d4-45d0-46d4-a580-34b0c50f83fd" (UID: "aaf405d4-45d0-46d4-a580-34b0c50f83fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 22:07:19.697825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.697798    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aaf405d4-45d0-46d4-a580-34b0c50f83fd" (UID: "aaf405d4-45d0-46d4-a580-34b0c50f83fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:07:19.698209 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.698188    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "aaf405d4-45d0-46d4-a580-34b0c50f83fd" (UID: "aaf405d4-45d0-46d4-a580-34b0c50f83fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:07:19.698209 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.698214    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-kube-api-access-47xzt" (OuterVolumeSpecName: "kube-api-access-47xzt") pod "aaf405d4-45d0-46d4-a580-34b0c50f83fd" (UID: "aaf405d4-45d0-46d4-a580-34b0c50f83fd"). InnerVolumeSpecName "kube-api-access-47xzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:07:19.798229 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.798193    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-envoy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:19.798374 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.798226    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:19.798374 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.798257    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47xzt\" (UniqueName: \"kubernetes.io/projected/aaf405d4-45d0-46d4-a580-34b0c50f83fd-kube-api-access-47xzt\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:19.798374 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:19.798272    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf405d4-45d0-46d4-a580-34b0c50f83fd-istio-podinfo\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:20.071869 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.071833    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt" event={"ID":"60058ec6-fcbd-4625-b93e-4c1d5f43a01a","Type":"ContainerStarted","Data":"15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028"}
Jul 31 22:07:20.072056 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.071898    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:20.072056 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.071918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt" event={"ID":"60058ec6-fcbd-4625-b93e-4c1d5f43a01a","Type":"ContainerStarted","Data":"5c0363c6c4e4ff6d0a7ca2398e0a5b6a4b0cb6229ebc79845a6f0d923cdf935f"}
Jul 31 22:07:20.073492 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.073468    2561 generic.go:356] "Generic (PLEG): container finished" podID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerID="9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a" exitCode=0
Jul 31 22:07:20.073592 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.073510    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"
Jul 31 22:07:20.073592 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.073516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" event={"ID":"aaf405d4-45d0-46d4-a580-34b0c50f83fd","Type":"ContainerDied","Data":"9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a"}
Jul 31 22:07:20.073592 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.073550    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc" event={"ID":"aaf405d4-45d0-46d4-a580-34b0c50f83fd","Type":"ContainerDied","Data":"6a7bd95c282bab256abbb04770ea0e305b9eebce0495743336c23f339714bd40"}
Jul 31 22:07:20.073592 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.073571    2561 scope.go:117] "RemoveContainer" containerID="9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a"
Jul 31 22:07:20.082620 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.082605    2561 scope.go:117] "RemoveContainer" containerID="e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79"
Jul 31 22:07:20.090126 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.090082    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt" podStartSLOduration=1.090065132 podStartE2EDuration="1.090065132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:07:20.087073027 +0000 UTC m=+1880.349821392" watchObservedRunningTime="2026-07-31 22:07:20.090065132 +0000 UTC m=+1880.352813501"
Jul 31 22:07:20.090977 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.090960    2561 scope.go:117] "RemoveContainer" containerID="9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a"
Jul 31 22:07:20.091264 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:07:20.091242    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a\": container with ID starting with 9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a not found: ID does not exist" containerID="9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a"
Jul 31 22:07:20.091334 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.091275    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a"} err="failed to get container status \"9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a\": rpc error: code = NotFound desc = could not find container \"9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a\": container with ID starting with 9afc5c5123d7bcd6eb70a0c5e6eb56ce2390f59c24da3d040a7c8828e974383a not found: ID does not exist"
Jul 31 22:07:20.091334 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.091297    2561 scope.go:117] "RemoveContainer" containerID="e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79"
Jul 31 22:07:20.091560 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:07:20.091542    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79\": container with ID starting with e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79 not found: ID does not exist" containerID="e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79"
Jul 31 22:07:20.091608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.091566    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79"} err="failed to get container status \"e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79\": rpc error: code = NotFound desc = could not find container \"e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79\": container with ID starting with e0eeed2063bd208355213995af821bd4d019b6f8309e86730ae93b58bdbefe79 not found: ID does not exist"
Jul 31 22:07:20.100566 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.100519    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"]
Jul 31 22:07:20.102341 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.102319    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8daf3c2f-openshift-default-7cbfb4798b-lgbqc"]
Jul 31 22:07:20.234668 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:20.234633    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" path="/var/lib/kubelet/pods/aaf405d4-45d0-46d4-a580-34b0c50f83fd/volumes"
Jul 31 22:07:31.082105 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:31.082072    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:32.060177 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:32.059155    2561 patch_prober.go:28] interesting pod/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:51904->10.132.0.51:15021: read: connection reset by peer" start-of-body=
Jul 31 22:07:32.060177 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:32.059219    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:51904->10.132.0.51:15021: read: connection reset by peer"
Jul 31 22:07:32.140565 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:32.140534    2561 generic.go:356] "Generic (PLEG): container finished" podID="08880c5c-b10f-442e-a542-f48922f03f7c" containerID="d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de" exitCode=0
Jul 31 22:07:32.140860 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:32.140605    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" event={"ID":"08880c5c-b10f-442e-a542-f48922f03f7c","Type":"ContainerDied","Data":"d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de"}
Jul 31 22:07:32.141016 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:32.141003    2561 scope.go:117] "RemoveContainer" containerID="d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de"
Jul 31 22:07:32.142009 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:32.141982    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:07:32.142073 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:32.142050    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:07:32.142112 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:32.142084    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:07:33.148040 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:33.148004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" event={"ID":"08880c5c-b10f-442e-a542-f48922f03f7c","Type":"ContainerStarted","Data":"76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04"}
Jul 31 22:07:33.337169 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:33.337113    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:33.337403 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:33.337380    2561 patch_prober.go:28] interesting pod/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 31 22:07:33.337455 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:33.337437    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 31 22:07:34.337178 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:34.337125    2561 patch_prober.go:28] interesting pod/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 31 22:07:34.337579 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:34.337204    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 31 22:07:35.337852 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:35.337814    2561 patch_prober.go:28] interesting pod/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 31 22:07:35.338251 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:35.337866    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 31 22:07:36.337265 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:36.337227    2561 patch_prober.go:28] interesting pod/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 31 22:07:36.337419 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:36.337284    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 31 22:07:37.337564 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:37.337522    2561 patch_prober.go:28] interesting pod/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 31 22:07:37.337973 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:37.337583    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 31 22:07:38.337657 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:38.337617    2561 patch_prober.go:28] interesting pod/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 31 22:07:38.338066 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:38.337688    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 31 22:07:39.340514 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:39.340478    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:39.340955 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:39.340883    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:39.341616 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:39.341597    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:49.167402 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:49.167368    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"]
Jul 31 22:07:49.167946 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:49.167598    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt" podUID="60058ec6-fcbd-4625-b93e-4c1d5f43a01a" containerName="payload-pre-processing" containerID="cri-o://15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028" gracePeriod=30
Jul 31 22:07:49.424861 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:49.424783    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:49.454472 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:49.454433    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"]
Jul 31 22:07:49.454934 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:49.454893    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy" containerID="cri-o://76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04" gracePeriod=30
Jul 31 22:07:49.569681 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:49.569652    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlb96\" (UniqueName: \"kubernetes.io/projected/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-kube-api-access-jlb96\") pod \"60058ec6-fcbd-4625-b93e-4c1d5f43a01a\" (UID: \"60058ec6-fcbd-4625-b93e-4c1d5f43a01a\") "
Jul 31 22:07:49.569855 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:49.569712    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-plugins-config-volume\") pod \"60058ec6-fcbd-4625-b93e-4c1d5f43a01a\" (UID: \"60058ec6-fcbd-4625-b93e-4c1d5f43a01a\") "
Jul 31 22:07:49.570079 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:49.570056    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60058ec6-fcbd-4625-b93e-4c1d5f43a01a" (UID: "60058ec6-fcbd-4625-b93e-4c1d5f43a01a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:07:49.571754 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:49.571733    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-kube-api-access-jlb96" (OuterVolumeSpecName: "kube-api-access-jlb96") pod "60058ec6-fcbd-4625-b93e-4c1d5f43a01a" (UID: "60058ec6-fcbd-4625-b93e-4c1d5f43a01a"). InnerVolumeSpecName "kube-api-access-jlb96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:07:49.670710 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:49.670676    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlb96\" (UniqueName: \"kubernetes.io/projected/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-kube-api-access-jlb96\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:49.670710 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:49.670705    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60058ec6-fcbd-4625-b93e-4c1d5f43a01a-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:50.237673 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:50.237635    2561 generic.go:356] "Generic (PLEG): container finished" podID="60058ec6-fcbd-4625-b93e-4c1d5f43a01a" containerID="15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028" exitCode=0
Jul 31 22:07:50.238163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:50.237690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt" event={"ID":"60058ec6-fcbd-4625-b93e-4c1d5f43a01a","Type":"ContainerDied","Data":"15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028"}
Jul 31 22:07:50.238163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:50.237705    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"
Jul 31 22:07:50.238163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:50.237733    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt" event={"ID":"60058ec6-fcbd-4625-b93e-4c1d5f43a01a","Type":"ContainerDied","Data":"5c0363c6c4e4ff6d0a7ca2398e0a5b6a4b0cb6229ebc79845a6f0d923cdf935f"}
Jul 31 22:07:50.238163 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:50.237763    2561 scope.go:117] "RemoveContainer" containerID="15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028"
Jul 31 22:07:50.249234 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:50.249216    2561 scope.go:117] "RemoveContainer" containerID="15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028"
Jul 31 22:07:50.249510 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:07:50.249483    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028\": container with ID starting with 15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028 not found: ID does not exist" containerID="15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028"
Jul 31 22:07:50.249598 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:50.249518    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028"} err="failed to get container status \"15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028\": rpc error: code = NotFound desc = could not find container \"15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028\": container with ID starting with 15261bbff9103b6321d39f99a1e0618b6875ef093fd9c94ed74ad387e075d028 not found: ID does not exist"
Jul 31 22:07:50.262628 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:50.262607    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"]
Jul 31 22:07:50.266550 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:50.266527    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3a03c45-7f9c86c49b-xw7nt"]
Jul 31 22:07:52.227681 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:52.227645    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60058ec6-fcbd-4625-b93e-4c1d5f43a01a" path="/var/lib/kubelet/pods/60058ec6-fcbd-4625-b93e-4c1d5f43a01a/volumes"
Jul 31 22:07:54.344031 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.343996    2561 patch_prober.go:28] interesting pod/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 22:07:54.344415 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.344060    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 22:07:54.716023 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.716000    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:54.817367 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817339    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-certs\") pod \"08880c5c-b10f-442e-a542-f48922f03f7c\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") "
Jul 31 22:07:54.817554 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817380    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-data\") pod \"08880c5c-b10f-442e-a542-f48922f03f7c\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") "
Jul 31 22:07:54.817554 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817448    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-socket\") pod \"08880c5c-b10f-442e-a542-f48922f03f7c\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") "
Jul 31 22:07:54.817554 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817472    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-envoy\") pod \"08880c5c-b10f-442e-a542-f48922f03f7c\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") "
Jul 31 22:07:54.817554 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817514    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-credential-socket\") pod \"08880c5c-b10f-442e-a542-f48922f03f7c\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") "
Jul 31 22:07:54.817554 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817531    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08880c5c-b10f-442e-a542-f48922f03f7c-istiod-ca-cert\") pod \"08880c5c-b10f-442e-a542-f48922f03f7c\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") "
Jul 31 22:07:54.817825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817560    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08880c5c-b10f-442e-a542-f48922f03f7c-istio-podinfo\") pod \"08880c5c-b10f-442e-a542-f48922f03f7c\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") "
Jul 31 22:07:54.817825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817589    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft9l5\" (UniqueName: \"kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-kube-api-access-ft9l5\") pod \"08880c5c-b10f-442e-a542-f48922f03f7c\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") "
Jul 31 22:07:54.817825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817618    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-istio-token\") pod \"08880c5c-b10f-442e-a542-f48922f03f7c\" (UID: \"08880c5c-b10f-442e-a542-f48922f03f7c\") "
Jul 31 22:07:54.817825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817613    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08880c5c-b10f-442e-a542-f48922f03f7c" (UID: "08880c5c-b10f-442e-a542-f48922f03f7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:07:54.817825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817702    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-data" (OuterVolumeSpecName: "istio-data") pod "08880c5c-b10f-442e-a542-f48922f03f7c" (UID: "08880c5c-b10f-442e-a542-f48922f03f7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:07:54.817825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817760    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08880c5c-b10f-442e-a542-f48922f03f7c" (UID: "08880c5c-b10f-442e-a542-f48922f03f7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:07:54.818148 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.817850    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08880c5c-b10f-442e-a542-f48922f03f7c" (UID: "08880c5c-b10f-442e-a542-f48922f03f7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:07:54.818148 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.818012    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:54.818148 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.818069    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-credential-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:54.818148 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.818083    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-workload-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:54.818148 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.818096    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-data\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:54.818392 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.818186    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08880c5c-b10f-442e-a542-f48922f03f7c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08880c5c-b10f-442e-a542-f48922f03f7c" (UID: "08880c5c-b10f-442e-a542-f48922f03f7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:07:54.819875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.819825    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08880c5c-b10f-442e-a542-f48922f03f7c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08880c5c-b10f-442e-a542-f48922f03f7c" (UID: "08880c5c-b10f-442e-a542-f48922f03f7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 22:07:54.820246 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.820196    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-istio-token" (OuterVolumeSpecName: "istio-token") pod "08880c5c-b10f-442e-a542-f48922f03f7c" (UID: "08880c5c-b10f-442e-a542-f48922f03f7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:07:54.820375 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.820293    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-kube-api-access-ft9l5" (OuterVolumeSpecName: "kube-api-access-ft9l5") pod "08880c5c-b10f-442e-a542-f48922f03f7c" (UID: "08880c5c-b10f-442e-a542-f48922f03f7c"). InnerVolumeSpecName "kube-api-access-ft9l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:07:54.820473 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.820455    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08880c5c-b10f-442e-a542-f48922f03f7c" (UID: "08880c5c-b10f-442e-a542-f48922f03f7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:07:54.919285 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.919200    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08880c5c-b10f-442e-a542-f48922f03f7c-istio-envoy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:54.919285 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.919227    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08880c5c-b10f-442e-a542-f48922f03f7c-istiod-ca-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:54.919285 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.919240    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08880c5c-b10f-442e-a542-f48922f03f7c-istio-podinfo\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:54.919285 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.919251    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft9l5\" (UniqueName: \"kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-kube-api-access-ft9l5\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:54.919285 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:54.919265    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08880c5c-b10f-442e-a542-f48922f03f7c-istio-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:07:55.265971 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.265885    2561 generic.go:356] "Generic (PLEG): container finished" podID="08880c5c-b10f-442e-a542-f48922f03f7c" containerID="76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04" exitCode=0
Jul 31 22:07:55.265971 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.265942    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"
Jul 31 22:07:55.266152 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.265965    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" event={"ID":"08880c5c-b10f-442e-a542-f48922f03f7c","Type":"ContainerDied","Data":"76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04"}
Jul 31 22:07:55.266152 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.266017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd" event={"ID":"08880c5c-b10f-442e-a542-f48922f03f7c","Type":"ContainerDied","Data":"bf55f610827d9170efc9770ae752ce99a1184a3e907bdccff47426cd1e728f27"}
Jul 31 22:07:55.266152 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.266039    2561 scope.go:117] "RemoveContainer" containerID="76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04"
Jul 31 22:07:55.275010 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.274983    2561 scope.go:117] "RemoveContainer" containerID="d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de"
Jul 31 22:07:55.282849 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.282834    2561 scope.go:117] "RemoveContainer" containerID="76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04"
Jul 31 22:07:55.283092 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:07:55.283076    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04\": container with ID starting with 76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04 not found: ID does not exist" containerID="76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04"
Jul 31 22:07:55.283174 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.283098    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04"} err="failed to get container status \"76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04\": rpc error: code = NotFound desc = could not find container \"76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04\": container with ID starting with 76c13d7f9e59fa949b90d115915a0dc0ba7852fb961d43d8b0049476d88e4c04 not found: ID does not exist"
Jul 31 22:07:55.283174 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.283113    2561 scope.go:117] "RemoveContainer" containerID="d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de"
Jul 31 22:07:55.283356 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:07:55.283340    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de\": container with ID starting with d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de not found: ID does not exist" containerID="d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de"
Jul 31 22:07:55.283397 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.283360    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de"} err="failed to get container status \"d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de\": rpc error: code = NotFound desc = could not find container \"d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de\": container with ID starting with d4eb1cbea8348831b7d575fcb0aca19af9eaf43d12e382cdc7f617d2fa6785de not found: ID does not exist"
Jul 31 22:07:55.294710 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.294690    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"]
Jul 31 22:07:55.313981 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:55.313959    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e3a03c45-openshift-default-79f46ff88-nlxkd"]
Jul 31 22:07:56.228440 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:07:56.228408    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" path="/var/lib/kubelet/pods/08880c5c-b10f-442e-a542-f48922f03f7c/volumes"
Jul 31 22:09:48.537238 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537202    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"]
Jul 31 22:09:48.537711 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537574    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy"
Jul 31 22:09:48.537711 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537586    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy"
Jul 31 22:09:48.537711 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537607    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy"
Jul 31 22:09:48.537711 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537613    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy"
Jul 31 22:09:48.537711 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537621    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60058ec6-fcbd-4625-b93e-4c1d5f43a01a" containerName="payload-pre-processing"
Jul 31 22:09:48.537711 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537627    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60058ec6-fcbd-4625-b93e-4c1d5f43a01a" containerName="payload-pre-processing"
Jul 31 22:09:48.537711 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537639    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy"
Jul 31 22:09:48.537711 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537644    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy"
Jul 31 22:09:48.537711 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537711    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy"
Jul 31 22:09:48.537983 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537720    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy"
Jul 31 22:09:48.537983 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537726    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy"
Jul 31 22:09:48.537983 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.537730    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60058ec6-fcbd-4625-b93e-4c1d5f43a01a" containerName="payload-pre-processing"
Jul 31 22:09:48.540453 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.540436    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.542765 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.542744    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-723a1787-openshift-default-dockercfg-qqnbb\""
Jul 31 22:09:48.542892 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.542744    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 22:09:48.573709 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.573680    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"]
Jul 31 22:09:48.629719 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.629682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-podinfo\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.629719 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.629723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjtgf\" (UniqueName: \"kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-kube-api-access-vjtgf\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.629929 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.629744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-certs\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.629929 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.629830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-envoy\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.629929 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.629870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-credential-socket\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.629929 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.629896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-socket\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.630061 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.629975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-data\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.630061 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.630007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istiod-ca-cert\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.630159 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.630065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-token\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731365 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-credential-socket\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731365 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-socket\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731651 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731403    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-data\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731651 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istiod-ca-cert\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731651 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-token\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731651 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-podinfo\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731651 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtgf\" (UniqueName: \"kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-kube-api-access-vjtgf\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731651 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-certs\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731938 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-credential-socket\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731938 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-envoy\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731938 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-socket\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.731938 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.731856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-data\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.732119 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.732094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-certs\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.732242 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.732227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istiod-ca-cert\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.733875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.733855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-envoy\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.733967 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.733915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-podinfo\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.739309 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.739286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-token\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.739502 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.739484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtgf\" (UniqueName: \"kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-kube-api-access-vjtgf\") pod \"e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") " pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.850862 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.850758    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:48.992091 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:48.992052    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"]
Jul 31 22:09:49.000673 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:09:49.000637    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110b7d8d_6916_4c81_a3d6_8557da6ef3fa.slice/crio-7962fb16fe89cd582b708bcb1cfed2f8a699c2f42231a62f2e545f6e20ff0cfe WatchSource:0}: Error finding container 7962fb16fe89cd582b708bcb1cfed2f8a699c2f42231a62f2e545f6e20ff0cfe: Status 404 returned error can't find the container with id 7962fb16fe89cd582b708bcb1cfed2f8a699c2f42231a62f2e545f6e20ff0cfe
Jul 31 22:09:49.002676 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:49.002641    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:09:49.002754 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:49.002709    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:09:49.002754 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:49.002737    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:09:49.808986 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:49.808950    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" event={"ID":"110b7d8d-6916-4c81-a3d6-8557da6ef3fa","Type":"ContainerStarted","Data":"fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22"}
Jul 31 22:09:49.809408 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:49.809003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" event={"ID":"110b7d8d-6916-4c81-a3d6-8557da6ef3fa","Type":"ContainerStarted","Data":"7962fb16fe89cd582b708bcb1cfed2f8a699c2f42231a62f2e545f6e20ff0cfe"}
Jul 31 22:09:49.826563 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:49.826506    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" podStartSLOduration=1.8264902790000002 podStartE2EDuration="1.826490279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:09:49.826045464 +0000 UTC m=+2030.088793829" watchObservedRunningTime="2026-07-31 22:09:49.826490279 +0000 UTC m=+2030.089238642"
Jul 31 22:09:49.851387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:49.851358    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:50.856302 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:50.856274    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:51.818525 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:51.818496    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:51.819583 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:51.819559    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:09:53.047869 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.047831    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"]
Jul 31 22:09:53.048432 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.048410    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy"
Jul 31 22:09:53.048487 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.048436    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaf405d4-45d0-46d4-a580-34b0c50f83fd" containerName="istio-proxy"
Jul 31 22:09:53.048581 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.048568    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="08880c5c-b10f-442e-a542-f48922f03f7c" containerName="istio-proxy"
Jul 31 22:09:53.052250 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.052229    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:53.055739 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.055716    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-723a1787-serving-cert\""
Jul 31 22:09:53.061239 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.061214    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"]
Jul 31 22:09:53.171115 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.171078    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khrwc\" (UniqueName: \"kubernetes.io/projected/0e79cc8a-1bf0-4136-9b02-901163733639-kube-api-access-khrwc\") pod \"maas-api-e2e-mt-723a1787-79dc97558f-xtg8v\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:53.171341 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.171209    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e79cc8a-1bf0-4136-9b02-901163733639-maas-api-tls\") pod \"maas-api-e2e-mt-723a1787-79dc97558f-xtg8v\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:53.171341 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.171253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e79cc8a-1bf0-4136-9b02-901163733639-openshift-service-ca\") pod \"maas-api-e2e-mt-723a1787-79dc97558f-xtg8v\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:53.272024 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.271988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e79cc8a-1bf0-4136-9b02-901163733639-maas-api-tls\") pod \"maas-api-e2e-mt-723a1787-79dc97558f-xtg8v\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:53.272283 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.272049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e79cc8a-1bf0-4136-9b02-901163733639-openshift-service-ca\") pod \"maas-api-e2e-mt-723a1787-79dc97558f-xtg8v\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:53.272283 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.272097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khrwc\" (UniqueName: \"kubernetes.io/projected/0e79cc8a-1bf0-4136-9b02-901163733639-kube-api-access-khrwc\") pod \"maas-api-e2e-mt-723a1787-79dc97558f-xtg8v\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:53.272748 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.272725    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e79cc8a-1bf0-4136-9b02-901163733639-openshift-service-ca\") pod \"maas-api-e2e-mt-723a1787-79dc97558f-xtg8v\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:53.274686 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.274667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e79cc8a-1bf0-4136-9b02-901163733639-maas-api-tls\") pod \"maas-api-e2e-mt-723a1787-79dc97558f-xtg8v\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:53.279914 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.279890    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khrwc\" (UniqueName: \"kubernetes.io/projected/0e79cc8a-1bf0-4136-9b02-901163733639-kube-api-access-khrwc\") pod \"maas-api-e2e-mt-723a1787-79dc97558f-xtg8v\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:53.363819 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.363722    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:53.512293 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.512116    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"]
Jul 31 22:09:53.514648 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:09:53.514619    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e79cc8a_1bf0_4136_9b02_901163733639.slice/crio-f8952030a14bdb5632725375023057c43c6b0e22143b1390d99c8ea608fb66f2 WatchSource:0}: Error finding container f8952030a14bdb5632725375023057c43c6b0e22143b1390d99c8ea608fb66f2: Status 404 returned error can't find the container with id f8952030a14bdb5632725375023057c43c6b0e22143b1390d99c8ea608fb66f2
Jul 31 22:09:53.752280 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.752199    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"]
Jul 31 22:09:53.756951 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.756929    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:09:53.759607 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.759586    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-723a1787-dockercfg-2kcw8\""
Jul 31 22:09:53.759732 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.759589    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-723a1787\""
Jul 31 22:09:53.771415 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.771384    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"]
Jul 31 22:09:53.827863 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.827816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v" event={"ID":"0e79cc8a-1bf0-4136-9b02-901163733639","Type":"ContainerStarted","Data":"f8952030a14bdb5632725375023057c43c6b0e22143b1390d99c8ea608fb66f2"}
Jul 31 22:09:53.877998 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.877963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2bk\" (UniqueName: \"kubernetes.io/projected/02fe9ad5-de02-48ba-be97-c7eec70e3a42-kube-api-access-xr2bk\") pod \"payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn\" (UID: \"02fe9ad5-de02-48ba-be97-c7eec70e3a42\") " pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:09:53.878241 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.878210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fe9ad5-de02-48ba-be97-c7eec70e3a42-plugins-config-volume\") pod \"payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn\" (UID: \"02fe9ad5-de02-48ba-be97-c7eec70e3a42\") " pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:09:53.979546 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.979507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fe9ad5-de02-48ba-be97-c7eec70e3a42-plugins-config-volume\") pod \"payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn\" (UID: \"02fe9ad5-de02-48ba-be97-c7eec70e3a42\") " pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:09:53.979751 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.979572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2bk\" (UniqueName: \"kubernetes.io/projected/02fe9ad5-de02-48ba-be97-c7eec70e3a42-kube-api-access-xr2bk\") pod \"payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn\" (UID: \"02fe9ad5-de02-48ba-be97-c7eec70e3a42\") " pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:09:53.980384 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.980361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fe9ad5-de02-48ba-be97-c7eec70e3a42-plugins-config-volume\") pod \"payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn\" (UID: \"02fe9ad5-de02-48ba-be97-c7eec70e3a42\") " pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:09:53.987220 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:53.987182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2bk\" (UniqueName: \"kubernetes.io/projected/02fe9ad5-de02-48ba-be97-c7eec70e3a42-kube-api-access-xr2bk\") pod \"payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn\" (UID: \"02fe9ad5-de02-48ba-be97-c7eec70e3a42\") " pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:09:54.069161 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:54.069093    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:09:54.219358 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:54.219331    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"]
Jul 31 22:09:54.222234 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:09:54.222207    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fe9ad5_de02_48ba_be97_c7eec70e3a42.slice/crio-ad0711a3f83515b6c4db192771e6dbad26e8384d494d228d63b6ba65d489104e WatchSource:0}: Error finding container ad0711a3f83515b6c4db192771e6dbad26e8384d494d228d63b6ba65d489104e: Status 404 returned error can't find the container with id ad0711a3f83515b6c4db192771e6dbad26e8384d494d228d63b6ba65d489104e
Jul 31 22:09:54.834700 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:54.834657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v" event={"ID":"0e79cc8a-1bf0-4136-9b02-901163733639","Type":"ContainerStarted","Data":"2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423"}
Jul 31 22:09:54.834914 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:54.834766    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:09:54.836343 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:54.836316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn" event={"ID":"02fe9ad5-de02-48ba-be97-c7eec70e3a42","Type":"ContainerStarted","Data":"13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78"}
Jul 31 22:09:54.836489 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:54.836362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn" event={"ID":"02fe9ad5-de02-48ba-be97-c7eec70e3a42","Type":"ContainerStarted","Data":"ad0711a3f83515b6c4db192771e6dbad26e8384d494d228d63b6ba65d489104e"}
Jul 31 22:09:54.836489 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:54.836447    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:09:54.869783 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:54.869731    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn" podStartSLOduration=1.8697160400000001 podStartE2EDuration="1.86971604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:09:54.86838134 +0000 UTC m=+2035.131129706" watchObservedRunningTime="2026-07-31 22:09:54.86971604 +0000 UTC m=+2035.132464430"
Jul 31 22:09:54.870839 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:09:54.870807    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v" podStartSLOduration=1.218181577 podStartE2EDuration="1.870796349s" totalImagesPullingTime="652.614772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:09:54.853247054 +0000 UTC m=+2035.115995420" watchObservedRunningTime="2026-07-31 22:09:54.870796349 +0000 UTC m=+2035.133544712"
Jul 31 22:10:00.847110 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:00.847080    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:10:05.843577 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:05.843546    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:10:05.894177 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:05.894108    2561 generic.go:356] "Generic (PLEG): container finished" podID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerID="fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22" exitCode=0
Jul 31 22:10:05.894373 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:05.894184    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" event={"ID":"110b7d8d-6916-4c81-a3d6-8557da6ef3fa","Type":"ContainerDied","Data":"fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22"}
Jul 31 22:10:05.894588 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:05.894573    2561 scope.go:117] "RemoveContainer" containerID="fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22"
Jul 31 22:10:05.895728 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:05.895696    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:10:05.895811 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:05.895772    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:10:05.895873 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:05.895819    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:10:06.818609 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:06.818573    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:10:06.905626 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:06.905578    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" event={"ID":"110b7d8d-6916-4c81-a3d6-8557da6ef3fa","Type":"ContainerStarted","Data":"c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55"}
Jul 31 22:10:07.851993 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:07.851952    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:10:07.853070 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:07.853043    2561 patch_prober.go:28] interesting pod/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 31 22:10:07.853181 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:07.853094    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 31 22:10:08.851765 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:08.851722    2561 patch_prober.go:28] interesting pod/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 31 22:10:08.852220 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:08.851798    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 31 22:10:09.852236 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:09.852197    2561 patch_prober.go:28] interesting pod/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 31 22:10:09.852642 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:09.852258    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 31 22:10:10.851266 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:10.851224    2561 patch_prober.go:28] interesting pod/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 31 22:10:10.851469 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:10.851295    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 31 22:10:11.851370 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:11.851333    2561 patch_prober.go:28] interesting pod/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 31 22:10:11.851817 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:11.851393    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 31 22:10:12.855964 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:12.855931    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:10:12.856363 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:12.856213    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:10:12.856986 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:12.856969    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:10:34.533380 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.533109    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"]
Jul 31 22:10:34.533955 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.533414    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v" podUID="0e79cc8a-1bf0-4136-9b02-901163733639" containerName="maas-api" containerID="cri-o://2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423" gracePeriod=30
Jul 31 22:10:34.591307 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.591262    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"]
Jul 31 22:10:34.591557 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.591526    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn" podUID="02fe9ad5-de02-48ba-be97-c7eec70e3a42" containerName="payload-processing" containerID="cri-o://13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78" gracePeriod=30
Jul 31 22:10:34.863588 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.863558    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:10:34.887238 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.887208    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:10:34.974845 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.974803    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e79cc8a-1bf0-4136-9b02-901163733639-openshift-service-ca\") pod \"0e79cc8a-1bf0-4136-9b02-901163733639\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") "
Jul 31 22:10:34.975023 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.974902    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e79cc8a-1bf0-4136-9b02-901163733639-maas-api-tls\") pod \"0e79cc8a-1bf0-4136-9b02-901163733639\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") "
Jul 31 22:10:34.975023 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.974925    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khrwc\" (UniqueName: \"kubernetes.io/projected/0e79cc8a-1bf0-4136-9b02-901163733639-kube-api-access-khrwc\") pod \"0e79cc8a-1bf0-4136-9b02-901163733639\" (UID: \"0e79cc8a-1bf0-4136-9b02-901163733639\") "
Jul 31 22:10:34.975217 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.975192    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e79cc8a-1bf0-4136-9b02-901163733639-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e79cc8a-1bf0-4136-9b02-901163733639" (UID: "0e79cc8a-1bf0-4136-9b02-901163733639"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:10:34.977065 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.977036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e79cc8a-1bf0-4136-9b02-901163733639-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e79cc8a-1bf0-4136-9b02-901163733639" (UID: "0e79cc8a-1bf0-4136-9b02-901163733639"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:10:34.977065 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:34.977059    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e79cc8a-1bf0-4136-9b02-901163733639-kube-api-access-khrwc" (OuterVolumeSpecName: "kube-api-access-khrwc") pod "0e79cc8a-1bf0-4136-9b02-901163733639" (UID: "0e79cc8a-1bf0-4136-9b02-901163733639"). InnerVolumeSpecName "kube-api-access-khrwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:10:35.047348 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.047313    2561 generic.go:356] "Generic (PLEG): container finished" podID="0e79cc8a-1bf0-4136-9b02-901163733639" containerID="2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423" exitCode=0
Jul 31 22:10:35.047536 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.047359    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"
Jul 31 22:10:35.047536 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.047405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v" event={"ID":"0e79cc8a-1bf0-4136-9b02-901163733639","Type":"ContainerDied","Data":"2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423"}
Jul 31 22:10:35.047536 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.047468    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v" event={"ID":"0e79cc8a-1bf0-4136-9b02-901163733639","Type":"ContainerDied","Data":"f8952030a14bdb5632725375023057c43c6b0e22143b1390d99c8ea608fb66f2"}
Jul 31 22:10:35.047536 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.047498    2561 scope.go:117] "RemoveContainer" containerID="2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423"
Jul 31 22:10:35.048800 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.048779    2561 generic.go:356] "Generic (PLEG): container finished" podID="02fe9ad5-de02-48ba-be97-c7eec70e3a42" containerID="13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78" exitCode=0
Jul 31 22:10:35.048869 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.048811    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn" event={"ID":"02fe9ad5-de02-48ba-be97-c7eec70e3a42","Type":"ContainerDied","Data":"13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78"}
Jul 31 22:10:35.048869 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.048829    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"
Jul 31 22:10:35.048869 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.048849    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn" event={"ID":"02fe9ad5-de02-48ba-be97-c7eec70e3a42","Type":"ContainerDied","Data":"ad0711a3f83515b6c4db192771e6dbad26e8384d494d228d63b6ba65d489104e"}
Jul 31 22:10:35.060558 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.058360    2561 scope.go:117] "RemoveContainer" containerID="2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423"
Jul 31 22:10:35.060558 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:10:35.058780    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423\": container with ID starting with 2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423 not found: ID does not exist" containerID="2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423"
Jul 31 22:10:35.060558 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.058820    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423"} err="failed to get container status \"2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423\": rpc error: code = NotFound desc = could not find container \"2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423\": container with ID starting with 2efb8d0b286dea83260114b2d7312a834a1af0829b4728952ae286b73f09b423 not found: ID does not exist"
Jul 31 22:10:35.060558 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.058846    2561 scope.go:117] "RemoveContainer" containerID="13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78"
Jul 31 22:10:35.068845 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.068819    2561 scope.go:117] "RemoveContainer" containerID="13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78"
Jul 31 22:10:35.069175 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:10:35.069152    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78\": container with ID starting with 13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78 not found: ID does not exist" containerID="13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78"
Jul 31 22:10:35.069281 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.069183    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78"} err="failed to get container status \"13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78\": rpc error: code = NotFound desc = could not find container \"13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78\": container with ID starting with 13e8a809bb69969327f0b9756a2a2bf8b772ee6f4b7a6dd2a2d7dc3dc4371d78 not found: ID does not exist"
Jul 31 22:10:35.076123 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.076096    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr2bk\" (UniqueName: \"kubernetes.io/projected/02fe9ad5-de02-48ba-be97-c7eec70e3a42-kube-api-access-xr2bk\") pod \"02fe9ad5-de02-48ba-be97-c7eec70e3a42\" (UID: \"02fe9ad5-de02-48ba-be97-c7eec70e3a42\") "
Jul 31 22:10:35.076260 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.076188    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"]
Jul 31 22:10:35.076260 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.076215    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fe9ad5-de02-48ba-be97-c7eec70e3a42-plugins-config-volume\") pod \"02fe9ad5-de02-48ba-be97-c7eec70e3a42\" (UID: \"02fe9ad5-de02-48ba-be97-c7eec70e3a42\") "
Jul 31 22:10:35.076404 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.076393    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e79cc8a-1bf0-4136-9b02-901163733639-maas-api-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:35.076442 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.076407    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khrwc\" (UniqueName: \"kubernetes.io/projected/0e79cc8a-1bf0-4136-9b02-901163733639-kube-api-access-khrwc\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:35.076442 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.076418    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e79cc8a-1bf0-4136-9b02-901163733639-openshift-service-ca\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:35.076561 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.076539    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fe9ad5-de02-48ba-be97-c7eec70e3a42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02fe9ad5-de02-48ba-be97-c7eec70e3a42" (UID: "02fe9ad5-de02-48ba-be97-c7eec70e3a42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:10:35.078473 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.078444    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fe9ad5-de02-48ba-be97-c7eec70e3a42-kube-api-access-xr2bk" (OuterVolumeSpecName: "kube-api-access-xr2bk") pod "02fe9ad5-de02-48ba-be97-c7eec70e3a42" (UID: "02fe9ad5-de02-48ba-be97-c7eec70e3a42"). InnerVolumeSpecName "kube-api-access-xr2bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:10:35.080403 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.080378    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723a1787-79dc97558f-xtg8v"]
Jul 31 22:10:35.177385 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.177339    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fe9ad5-de02-48ba-be97-c7eec70e3a42-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:35.177385 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.177377    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr2bk\" (UniqueName: \"kubernetes.io/projected/02fe9ad5-de02-48ba-be97-c7eec70e3a42-kube-api-access-xr2bk\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:35.374171 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.374114    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"]
Jul 31 22:10:35.378707 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:35.378671    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-723a1787-7788669bfd-t5lfn"]
Jul 31 22:10:36.227780 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:36.227743    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02fe9ad5-de02-48ba-be97-c7eec70e3a42" path="/var/lib/kubelet/pods/02fe9ad5-de02-48ba-be97-c7eec70e3a42/volumes"
Jul 31 22:10:36.228212 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:36.228191    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e79cc8a-1bf0-4136-9b02-901163733639" path="/var/lib/kubelet/pods/0e79cc8a-1bf0-4136-9b02-901163733639/volumes"
Jul 31 22:10:41.245837 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:41.245799    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"]
Jul 31 22:10:41.246305 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:41.246073    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy" containerID="cri-o://c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55" gracePeriod=30
Jul 31 22:10:42.859115 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:42.859066    2561 patch_prober.go:28] interesting pod/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 22:10:42.859544 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:42.859175    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 22:10:46.502845 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.502810    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:10:46.566626 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566592    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-credential-socket\") pod \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") "
Jul 31 22:10:46.566626 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566632    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-token\") pod \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") "
Jul 31 22:10:46.566927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566653    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-certs\") pod \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") "
Jul 31 22:10:46.566927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566668    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-socket\") pod \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") "
Jul 31 22:10:46.566927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566702    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-data\") pod \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") "
Jul 31 22:10:46.566927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566736    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjtgf\" (UniqueName: \"kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-kube-api-access-vjtgf\") pod \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") "
Jul 31 22:10:46.566927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566761    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istiod-ca-cert\") pod \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") "
Jul 31 22:10:46.566927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566793    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-envoy\") pod \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") "
Jul 31 22:10:46.566927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566857    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-podinfo\") pod \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\" (UID: \"110b7d8d-6916-4c81-a3d6-8557da6ef3fa\") "
Jul 31 22:10:46.567330 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566936    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "110b7d8d-6916-4c81-a3d6-8557da6ef3fa" (UID: "110b7d8d-6916-4c81-a3d6-8557da6ef3fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:10:46.567330 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.567320    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "110b7d8d-6916-4c81-a3d6-8557da6ef3fa" (UID: "110b7d8d-6916-4c81-a3d6-8557da6ef3fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:10:46.567435 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566982    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "110b7d8d-6916-4c81-a3d6-8557da6ef3fa" (UID: "110b7d8d-6916-4c81-a3d6-8557da6ef3fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:10:46.567435 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.566995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "110b7d8d-6916-4c81-a3d6-8557da6ef3fa" (UID: "110b7d8d-6916-4c81-a3d6-8557da6ef3fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:10:46.567539 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.567514    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "110b7d8d-6916-4c81-a3d6-8557da6ef3fa" (UID: "110b7d8d-6916-4c81-a3d6-8557da6ef3fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:10:46.569110 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.569074    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-kube-api-access-vjtgf" (OuterVolumeSpecName: "kube-api-access-vjtgf") pod "110b7d8d-6916-4c81-a3d6-8557da6ef3fa" (UID: "110b7d8d-6916-4c81-a3d6-8557da6ef3fa"). InnerVolumeSpecName "kube-api-access-vjtgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:10:46.569269 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.569157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "110b7d8d-6916-4c81-a3d6-8557da6ef3fa" (UID: "110b7d8d-6916-4c81-a3d6-8557da6ef3fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 22:10:46.569335 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.569319    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "110b7d8d-6916-4c81-a3d6-8557da6ef3fa" (UID: "110b7d8d-6916-4c81-a3d6-8557da6ef3fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:10:46.569449 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.569426    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "110b7d8d-6916-4c81-a3d6-8557da6ef3fa" (UID: "110b7d8d-6916-4c81-a3d6-8557da6ef3fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:10:46.668321 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.668285    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-credential-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:46.668321 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.668316    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:46.668321 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.668330    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:46.668561 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.668342    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-workload-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:46.668561 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.668350    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-data\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:46.668561 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.668359    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjtgf\" (UniqueName: \"kubernetes.io/projected/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-kube-api-access-vjtgf\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:46.668561 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.668367    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istiod-ca-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:46.668561 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.668376    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-envoy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:46.668561 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:46.668385    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110b7d8d-6916-4c81-a3d6-8557da6ef3fa-istio-podinfo\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:10:47.110726 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.110686    2561 generic.go:356] "Generic (PLEG): container finished" podID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerID="c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55" exitCode=0
Jul 31 22:10:47.110726 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.110728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" event={"ID":"110b7d8d-6916-4c81-a3d6-8557da6ef3fa","Type":"ContainerDied","Data":"c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55"}
Jul 31 22:10:47.110964 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.110731    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"
Jul 31 22:10:47.110964 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.110779    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m" event={"ID":"110b7d8d-6916-4c81-a3d6-8557da6ef3fa","Type":"ContainerDied","Data":"7962fb16fe89cd582b708bcb1cfed2f8a699c2f42231a62f2e545f6e20ff0cfe"}
Jul 31 22:10:47.110964 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.110802    2561 scope.go:117] "RemoveContainer" containerID="c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55"
Jul 31 22:10:47.120159 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.120115    2561 scope.go:117] "RemoveContainer" containerID="fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22"
Jul 31 22:10:47.131916 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.131892    2561 scope.go:117] "RemoveContainer" containerID="c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55"
Jul 31 22:10:47.132292 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:10:47.132270    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55\": container with ID starting with c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55 not found: ID does not exist" containerID="c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55"
Jul 31 22:10:47.132365 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.132302    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55"} err="failed to get container status \"c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55\": rpc error: code = NotFound desc = could not find container \"c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55\": container with ID starting with c953c64dd2ebe4abda6d9673ad9648d365f16a8a689734947b42e1c5822bef55 not found: ID does not exist"
Jul 31 22:10:47.132365 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.132320    2561 scope.go:117] "RemoveContainer" containerID="fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22"
Jul 31 22:10:47.132365 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.132343    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"]
Jul 31 22:10:47.132610 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:10:47.132592    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22\": container with ID starting with fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22 not found: ID does not exist" containerID="fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22"
Jul 31 22:10:47.132675 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.132621    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22"} err="failed to get container status \"fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22\": rpc error: code = NotFound desc = could not find container \"fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22\": container with ID starting with fbb717cac8fac56a037f3526a25a55a92e896f0979afdd475c2d650f53994d22 not found: ID does not exist"
Jul 31 22:10:47.135875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:47.135848    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-723a1787-openshift-default-7b496cdd8-5nz6m"]
Jul 31 22:10:48.233474 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:10:48.233437    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" path="/var/lib/kubelet/pods/110b7d8d-6916-4c81-a3d6-8557da6ef3fa/volumes"
Jul 31 22:11:00.955478 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:11:00.955433    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:11:00.957646 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:11:00.957619    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:12:57.000108 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.000065    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"]
Jul 31 22:12:57.000734 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.000630    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02fe9ad5-de02-48ba-be97-c7eec70e3a42" containerName="payload-processing"
Jul 31 22:12:57.000734 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.000652    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="02fe9ad5-de02-48ba-be97-c7eec70e3a42" containerName="payload-processing"
Jul 31 22:12:57.000734 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.000672    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy"
Jul 31 22:12:57.000734 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.000681    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy"
Jul 31 22:12:57.000734 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.000717    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e79cc8a-1bf0-4136-9b02-901163733639" containerName="maas-api"
Jul 31 22:12:57.000734 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.000725    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e79cc8a-1bf0-4136-9b02-901163733639" containerName="maas-api"
Jul 31 22:12:57.001045 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.000808    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy"
Jul 31 22:12:57.001045 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.000821    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e79cc8a-1bf0-4136-9b02-901163733639" containerName="maas-api"
Jul 31 22:12:57.001045 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.000831    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="02fe9ad5-de02-48ba-be97-c7eec70e3a42" containerName="payload-processing"
Jul 31 22:12:57.004088 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.004065    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.006265 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.006242    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 22:12:57.006444 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.006253    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b0a39ff7-openshift-default-dockercfg-pfwkd\""
Jul 31 22:12:57.013592 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.013568    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"]
Jul 31 22:12:57.134759 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.134704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-socket\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.134759 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.134756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-certs\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.134993 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.134782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-credential-socket\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.134993 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.134882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-data\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.134993 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.134911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae66016-70ae-4239-9650-13a6c116d0c8-istio-podinfo\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.134993 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.134930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae66016-70ae-4239-9650-13a6c116d0c8-istiod-ca-cert\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.135184 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.134993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-envoy\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.135184 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.135042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slqxj\" (UniqueName: \"kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-kube-api-access-slqxj\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.135184 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.135091    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-istio-token\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.236566 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.236518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-data\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.236566 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.236568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae66016-70ae-4239-9650-13a6c116d0c8-istio-podinfo\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.236843 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.236704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae66016-70ae-4239-9650-13a6c116d0c8-istiod-ca-cert\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.236843 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.236765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-envoy\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.236843 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.236798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slqxj\" (UniqueName: \"kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-kube-api-access-slqxj\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.236999 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.236827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-istio-token\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.237205 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.237181    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-socket\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.237312 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.237220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-certs\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.237312 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.237257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-credential-socket\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.237569 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.237537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-data\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.237665 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.237576    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae66016-70ae-4239-9650-13a6c116d0c8-istiod-ca-cert\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.237719 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.237671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-socket\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.237719 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.237686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-certs\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.237792 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.237731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-credential-socket\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.239106 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.239082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-envoy\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.239255 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.239238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae66016-70ae-4239-9650-13a6c116d0c8-istio-podinfo\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.244829 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.244796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slqxj\" (UniqueName: \"kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-kube-api-access-slqxj\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.244919 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.244804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-istio-token\") pod \"e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") " pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.316100 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.316063    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:57.467729 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.467702    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"]
Jul 31 22:12:57.469899 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:12:57.469870    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae66016_70ae_4239_9650_13a6c116d0c8.slice/crio-35d0f22646fea15184c55d6d93ffd774e4d4aeecdbdc97b5240081da6f8cb38b WatchSource:0}: Error finding container 35d0f22646fea15184c55d6d93ffd774e4d4aeecdbdc97b5240081da6f8cb38b: Status 404 returned error can't find the container with id 35d0f22646fea15184c55d6d93ffd774e4d4aeecdbdc97b5240081da6f8cb38b
Jul 31 22:12:57.472759 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.472715    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:12:57.472860 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.472817    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:12:57.472920 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.472864    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:12:57.727578 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.727479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" event={"ID":"dae66016-70ae-4239-9650-13a6c116d0c8","Type":"ContainerStarted","Data":"652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428"}
Jul 31 22:12:57.727578 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.727556    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" event={"ID":"dae66016-70ae-4239-9650-13a6c116d0c8","Type":"ContainerStarted","Data":"35d0f22646fea15184c55d6d93ffd774e4d4aeecdbdc97b5240081da6f8cb38b"}
Jul 31 22:12:57.746860 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:57.746798    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" podStartSLOduration=1.746775958 podStartE2EDuration="1.746775958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:12:57.74537907 +0000 UTC m=+2218.008127434" watchObservedRunningTime="2026-07-31 22:12:57.746775958 +0000 UTC m=+2218.009524323"
Jul 31 22:12:58.317109 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:58.317072    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:58.318849 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:58.318811    2561 patch_prober.go:28] interesting pod/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 31 22:12:58.318993 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:58.318871    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 31 22:12:59.322858 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:59.322812    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:59.737373 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:59.737278    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:12:59.738282 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:12:59.738258    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:13:00.930522 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:00.930481    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"]
Jul 31 22:13:00.931119 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:00.931091    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy"
Jul 31 22:13:00.931119 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:00.931112    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy"
Jul 31 22:13:00.931362 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:00.931242    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="110b7d8d-6916-4c81-a3d6-8557da6ef3fa" containerName="istio-proxy"
Jul 31 22:13:00.934876 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:00.934850    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:00.936949 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:00.936925    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b0a39ff7-serving-cert\""
Jul 31 22:13:00.941689 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:00.941652    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"]
Jul 31 22:13:01.080656 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.080616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcs7j\" (UniqueName: \"kubernetes.io/projected/c3abb5d2-1386-408c-bce4-2b4e82aa4736-kube-api-access-pcs7j\") pod \"maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:01.080841 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.080700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3abb5d2-1386-408c-bce4-2b4e82aa4736-maas-api-tls\") pod \"maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:01.080841 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.080783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3abb5d2-1386-408c-bce4-2b4e82aa4736-openshift-service-ca\") pod \"maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:01.181559 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.181449    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3abb5d2-1386-408c-bce4-2b4e82aa4736-openshift-service-ca\") pod \"maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:01.181750 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.181623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcs7j\" (UniqueName: \"kubernetes.io/projected/c3abb5d2-1386-408c-bce4-2b4e82aa4736-kube-api-access-pcs7j\") pod \"maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:01.181750 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.181663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3abb5d2-1386-408c-bce4-2b4e82aa4736-maas-api-tls\") pod \"maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:01.181868 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:13:01.181776    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-serving-cert: secret "maas-api-e2e-mt-b0a39ff7-serving-cert" not found
Jul 31 22:13:01.181868 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:13:01.181837    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3abb5d2-1386-408c-bce4-2b4e82aa4736-maas-api-tls podName:c3abb5d2-1386-408c-bce4-2b4e82aa4736 nodeName:}" failed. No retries permitted until 2026-07-31 22:13:01.681817684 +0000 UTC m=+2221.944566046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3abb5d2-1386-408c-bce4-2b4e82aa4736-maas-api-tls") pod "maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6" (UID: "c3abb5d2-1386-408c-bce4-2b4e82aa4736") : secret "maas-api-e2e-mt-b0a39ff7-serving-cert" not found
Jul 31 22:13:01.182299 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.182274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3abb5d2-1386-408c-bce4-2b4e82aa4736-openshift-service-ca\") pod \"maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:01.190632 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.190605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcs7j\" (UniqueName: \"kubernetes.io/projected/c3abb5d2-1386-408c-bce4-2b4e82aa4736-kube-api-access-pcs7j\") pod \"maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:01.574014 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.573972    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"]
Jul 31 22:13:01.577733 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.577713    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:13:01.579900 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.579876    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b0a39ff7\""
Jul 31 22:13:01.580026 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.579946    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b0a39ff7-dockercfg-wj4v8\""
Jul 31 22:13:01.588279 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.588247    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"]
Jul 31 22:13:01.687230 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.687180    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedbd439-0d07-4b19-83f4-7c093ece9b44-plugins-config-volume\") pod \"payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp\" (UID: \"bedbd439-0d07-4b19-83f4-7c093ece9b44\") " pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:13:01.687423 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.687258    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6txj\" (UniqueName: \"kubernetes.io/projected/bedbd439-0d07-4b19-83f4-7c093ece9b44-kube-api-access-f6txj\") pod \"payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp\" (UID: \"bedbd439-0d07-4b19-83f4-7c093ece9b44\") " pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:13:01.687423 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.687289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3abb5d2-1386-408c-bce4-2b4e82aa4736-maas-api-tls\") pod \"maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:01.689794 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.689765    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3abb5d2-1386-408c-bce4-2b4e82aa4736-maas-api-tls\") pod \"maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:01.788189 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.788148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedbd439-0d07-4b19-83f4-7c093ece9b44-plugins-config-volume\") pod \"payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp\" (UID: \"bedbd439-0d07-4b19-83f4-7c093ece9b44\") " pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:13:01.788404 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.788228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6txj\" (UniqueName: \"kubernetes.io/projected/bedbd439-0d07-4b19-83f4-7c093ece9b44-kube-api-access-f6txj\") pod \"payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp\" (UID: \"bedbd439-0d07-4b19-83f4-7c093ece9b44\") " pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:13:01.788927 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.788904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedbd439-0d07-4b19-83f4-7c093ece9b44-plugins-config-volume\") pod \"payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp\" (UID: \"bedbd439-0d07-4b19-83f4-7c093ece9b44\") " pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:13:01.796277 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.796242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6txj\" (UniqueName: \"kubernetes.io/projected/bedbd439-0d07-4b19-83f4-7c093ece9b44-kube-api-access-f6txj\") pod \"payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp\" (UID: \"bedbd439-0d07-4b19-83f4-7c093ece9b44\") " pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:13:01.849078 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.848976    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:01.890274 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:01.890237    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:13:02.000834 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:02.000797    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"]
Jul 31 22:13:02.003190 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:13:02.003160    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3abb5d2_1386_408c_bce4_2b4e82aa4736.slice/crio-99735a6f48a4ea29cdb5265159af1de2d7b4bdf6de9d4b6c527ced9272699fd9 WatchSource:0}: Error finding container 99735a6f48a4ea29cdb5265159af1de2d7b4bdf6de9d4b6c527ced9272699fd9: Status 404 returned error can't find the container with id 99735a6f48a4ea29cdb5265159af1de2d7b4bdf6de9d4b6c527ced9272699fd9
Jul 31 22:13:02.004954 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:02.004930    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 22:13:02.043637 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:02.043605    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"]
Jul 31 22:13:02.046676 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:13:02.046637    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedbd439_0d07_4b19_83f4_7c093ece9b44.slice/crio-b0a04fd297c13715225389bad21f48157c1afeca9026cf8b75865ee5926ddc80 WatchSource:0}: Error finding container b0a04fd297c13715225389bad21f48157c1afeca9026cf8b75865ee5926ddc80: Status 404 returned error can't find the container with id b0a04fd297c13715225389bad21f48157c1afeca9026cf8b75865ee5926ddc80
Jul 31 22:13:02.755197 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:02.755155    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6" event={"ID":"c3abb5d2-1386-408c-bce4-2b4e82aa4736","Type":"ContainerStarted","Data":"0ace32a8887bba335f1e188108220646751d1330dd2bbe78e25fdaee8811a670"}
Jul 31 22:13:02.755458 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:02.755315    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:02.755458 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:02.755377    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6" event={"ID":"c3abb5d2-1386-408c-bce4-2b4e82aa4736","Type":"ContainerStarted","Data":"99735a6f48a4ea29cdb5265159af1de2d7b4bdf6de9d4b6c527ced9272699fd9"}
Jul 31 22:13:02.756945 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:02.756916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp" event={"ID":"bedbd439-0d07-4b19-83f4-7c093ece9b44","Type":"ContainerStarted","Data":"6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7"}
Jul 31 22:13:02.757075 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:02.756964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp" event={"ID":"bedbd439-0d07-4b19-83f4-7c093ece9b44","Type":"ContainerStarted","Data":"b0a04fd297c13715225389bad21f48157c1afeca9026cf8b75865ee5926ddc80"}
Jul 31 22:13:02.757075 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:02.757042    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:13:02.771238 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:02.771182    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6" podStartSLOduration=2.203096675 podStartE2EDuration="2.771163252s" totalImagesPullingTime="568.066577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:13:02.769808717 +0000 UTC m=+2223.032557081" watchObservedRunningTime="2026-07-31 22:13:02.771163252 +0000 UTC m=+2223.033911616"
Jul 31 22:13:02.786686 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:02.786614    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp" podStartSLOduration=1.786594316 podStartE2EDuration="1.786594316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:13:02.783963178 +0000 UTC m=+2223.046711537" watchObservedRunningTime="2026-07-31 22:13:02.786594316 +0000 UTC m=+2223.049342679"
Jul 31 22:13:08.768319 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:08.768288    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:13:13.765346 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:13.765305    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:13:13.819940 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:13.819906    2561 generic.go:356] "Generic (PLEG): container finished" podID="dae66016-70ae-4239-9650-13a6c116d0c8" containerID="652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428" exitCode=0
Jul 31 22:13:13.820108 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:13.819973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" event={"ID":"dae66016-70ae-4239-9650-13a6c116d0c8","Type":"ContainerDied","Data":"652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428"}
Jul 31 22:13:13.820393 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:13.820377    2561 scope.go:117] "RemoveContainer" containerID="652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428"
Jul 31 22:13:13.821445 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:13.821415    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:13:13.821529 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:13.821494    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:13:13.821589 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:13.821533    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:13:14.737460 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:14.737418    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:13:14.826965 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:14.826929    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" event={"ID":"dae66016-70ae-4239-9650-13a6c116d0c8","Type":"ContainerStarted","Data":"8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6"}
Jul 31 22:13:15.316734 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:15.316695    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:13:15.317680 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:15.317649    2561 patch_prober.go:28] interesting pod/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 31 22:13:15.317834 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:15.317710    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 31 22:13:16.317065 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:16.317022    2561 patch_prober.go:28] interesting pod/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 31 22:13:16.317502 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:16.317097    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 31 22:13:17.316660 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:17.316615    2561 patch_prober.go:28] interesting pod/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 31 22:13:17.316855 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:17.316690    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 31 22:13:18.316858 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:18.316818    2561 patch_prober.go:28] interesting pod/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 31 22:13:18.317295 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:18.316890    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 31 22:13:19.316557 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:19.316518    2561 patch_prober.go:28] interesting pod/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 31 22:13:19.316741 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:19.316578    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 31 22:13:20.322271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:20.322224    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:13:20.322751 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:20.322621    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:13:20.323540 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:13:20.323513    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:14:14.954400 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:14.954346    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"]
Jul 31 22:14:14.955318 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:14.955250    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6" podUID="c3abb5d2-1386-408c-bce4-2b4e82aa4736" containerName="maas-api" containerID="cri-o://0ace32a8887bba335f1e188108220646751d1330dd2bbe78e25fdaee8811a670" gracePeriod=30
Jul 31 22:14:15.009963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.009930    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"]
Jul 31 22:14:15.010309 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.010226    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp" podUID="bedbd439-0d07-4b19-83f4-7c093ece9b44" containerName="payload-processing" containerID="cri-o://6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7" gracePeriod=30
Jul 31 22:14:15.120380 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.120348    2561 generic.go:356] "Generic (PLEG): container finished" podID="c3abb5d2-1386-408c-bce4-2b4e82aa4736" containerID="0ace32a8887bba335f1e188108220646751d1330dd2bbe78e25fdaee8811a670" exitCode=0
Jul 31 22:14:15.120498 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.120424    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6" event={"ID":"c3abb5d2-1386-408c-bce4-2b4e82aa4736","Type":"ContainerDied","Data":"0ace32a8887bba335f1e188108220646751d1330dd2bbe78e25fdaee8811a670"}
Jul 31 22:14:15.268693 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.268665    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:14:15.297498 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.297470    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:14:15.380311 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.380274    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcs7j\" (UniqueName: \"kubernetes.io/projected/c3abb5d2-1386-408c-bce4-2b4e82aa4736-kube-api-access-pcs7j\") pod \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") "
Jul 31 22:14:15.380485 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.380320    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedbd439-0d07-4b19-83f4-7c093ece9b44-plugins-config-volume\") pod \"bedbd439-0d07-4b19-83f4-7c093ece9b44\" (UID: \"bedbd439-0d07-4b19-83f4-7c093ece9b44\") "
Jul 31 22:14:15.380485 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.380344    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6txj\" (UniqueName: \"kubernetes.io/projected/bedbd439-0d07-4b19-83f4-7c093ece9b44-kube-api-access-f6txj\") pod \"bedbd439-0d07-4b19-83f4-7c093ece9b44\" (UID: \"bedbd439-0d07-4b19-83f4-7c093ece9b44\") "
Jul 31 22:14:15.380485 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.380415    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3abb5d2-1386-408c-bce4-2b4e82aa4736-maas-api-tls\") pod \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") "
Jul 31 22:14:15.380485 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.380451    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3abb5d2-1386-408c-bce4-2b4e82aa4736-openshift-service-ca\") pod \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\" (UID: \"c3abb5d2-1386-408c-bce4-2b4e82aa4736\") "
Jul 31 22:14:15.380799 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.380766    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedbd439-0d07-4b19-83f4-7c093ece9b44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bedbd439-0d07-4b19-83f4-7c093ece9b44" (UID: "bedbd439-0d07-4b19-83f4-7c093ece9b44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:14:15.380957 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.380924    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3abb5d2-1386-408c-bce4-2b4e82aa4736-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3abb5d2-1386-408c-bce4-2b4e82aa4736" (UID: "c3abb5d2-1386-408c-bce4-2b4e82aa4736"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:14:15.382564 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.382536    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3abb5d2-1386-408c-bce4-2b4e82aa4736-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3abb5d2-1386-408c-bce4-2b4e82aa4736" (UID: "c3abb5d2-1386-408c-bce4-2b4e82aa4736"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:14:15.382672 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.382598    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedbd439-0d07-4b19-83f4-7c093ece9b44-kube-api-access-f6txj" (OuterVolumeSpecName: "kube-api-access-f6txj") pod "bedbd439-0d07-4b19-83f4-7c093ece9b44" (UID: "bedbd439-0d07-4b19-83f4-7c093ece9b44"). InnerVolumeSpecName "kube-api-access-f6txj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:14:15.382672 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.382628    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3abb5d2-1386-408c-bce4-2b4e82aa4736-kube-api-access-pcs7j" (OuterVolumeSpecName: "kube-api-access-pcs7j") pod "c3abb5d2-1386-408c-bce4-2b4e82aa4736" (UID: "c3abb5d2-1386-408c-bce4-2b4e82aa4736"). InnerVolumeSpecName "kube-api-access-pcs7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:14:15.481364 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.481255    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedbd439-0d07-4b19-83f4-7c093ece9b44-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:14:15.481364 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.481292    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6txj\" (UniqueName: \"kubernetes.io/projected/bedbd439-0d07-4b19-83f4-7c093ece9b44-kube-api-access-f6txj\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:14:15.481364 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.481306    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3abb5d2-1386-408c-bce4-2b4e82aa4736-maas-api-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:14:15.481364 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.481318    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3abb5d2-1386-408c-bce4-2b4e82aa4736-openshift-service-ca\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:14:15.481364 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:15.481330    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcs7j\" (UniqueName: \"kubernetes.io/projected/c3abb5d2-1386-408c-bce4-2b4e82aa4736-kube-api-access-pcs7j\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:14:16.126287 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.126259    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"
Jul 31 22:14:16.126287 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.126287    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6" event={"ID":"c3abb5d2-1386-408c-bce4-2b4e82aa4736","Type":"ContainerDied","Data":"99735a6f48a4ea29cdb5265159af1de2d7b4bdf6de9d4b6c527ced9272699fd9"}
Jul 31 22:14:16.126775 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.126352    2561 scope.go:117] "RemoveContainer" containerID="0ace32a8887bba335f1e188108220646751d1330dd2bbe78e25fdaee8811a670"
Jul 31 22:14:16.127681 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.127562    2561 generic.go:356] "Generic (PLEG): container finished" podID="bedbd439-0d07-4b19-83f4-7c093ece9b44" containerID="6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7" exitCode=0
Jul 31 22:14:16.127681 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.127607    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"
Jul 31 22:14:16.127681 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.127644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp" event={"ID":"bedbd439-0d07-4b19-83f4-7c093ece9b44","Type":"ContainerDied","Data":"6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7"}
Jul 31 22:14:16.127817 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.127683    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp" event={"ID":"bedbd439-0d07-4b19-83f4-7c093ece9b44","Type":"ContainerDied","Data":"b0a04fd297c13715225389bad21f48157c1afeca9026cf8b75865ee5926ddc80"}
Jul 31 22:14:16.135428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.135413    2561 scope.go:117] "RemoveContainer" containerID="6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7"
Jul 31 22:14:16.148222 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.148196    2561 scope.go:117] "RemoveContainer" containerID="6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7"
Jul 31 22:14:16.148576 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:14:16.148508    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7\": container with ID starting with 6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7 not found: ID does not exist" containerID="6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7"
Jul 31 22:14:16.148576 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.148536    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7"} err="failed to get container status \"6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7\": rpc error: code = NotFound desc = could not find container \"6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7\": container with ID starting with 6fcbafb6d8703ecf6b7bde6fea4833c49bf939d129150b5ab740499cffb7f5e7 not found: ID does not exist"
Jul 31 22:14:16.151185 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.151161    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"]
Jul 31 22:14:16.157467 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.157443    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0a39ff7-6bc6c87579-j6bl6"]
Jul 31 22:14:16.167262 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.167241    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"]
Jul 31 22:14:16.170455 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.170435    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0a39ff7-749db44bdb-rcdtp"]
Jul 31 22:14:16.227851 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.227758    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bedbd439-0d07-4b19-83f4-7c093ece9b44" path="/var/lib/kubelet/pods/bedbd439-0d07-4b19-83f4-7c093ece9b44/volumes"
Jul 31 22:14:16.228294 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:14:16.228271    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3abb5d2-1386-408c-bce4-2b4e82aa4736" path="/var/lib/kubelet/pods/c3abb5d2-1386-408c-bce4-2b4e82aa4736/volumes"
Jul 31 22:15:06.308125 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:06.308075    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"]
Jul 31 22:15:06.308766 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:06.308360    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy" containerID="cri-o://8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6" gracePeriod=30
Jul 31 22:15:11.558714 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.558681    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:15:11.602824 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.602720    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-socket\") pod \"dae66016-70ae-4239-9650-13a6c116d0c8\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") "
Jul 31 22:15:11.602824 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.602762    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-certs\") pod \"dae66016-70ae-4239-9650-13a6c116d0c8\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") "
Jul 31 22:15:11.602824 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.602792    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slqxj\" (UniqueName: \"kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-kube-api-access-slqxj\") pod \"dae66016-70ae-4239-9650-13a6c116d0c8\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") "
Jul 31 22:15:11.603127 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.602862    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-data\") pod \"dae66016-70ae-4239-9650-13a6c116d0c8\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") "
Jul 31 22:15:11.603127 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.602974    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-istio-token\") pod \"dae66016-70ae-4239-9650-13a6c116d0c8\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") "
Jul 31 22:15:11.603127 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-credential-socket\") pod \"dae66016-70ae-4239-9650-13a6c116d0c8\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") "
Jul 31 22:15:11.603127 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603042    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae66016-70ae-4239-9650-13a6c116d0c8-istio-podinfo\") pod \"dae66016-70ae-4239-9650-13a6c116d0c8\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") "
Jul 31 22:15:11.603127 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603087    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae66016-70ae-4239-9650-13a6c116d0c8-istiod-ca-cert\") pod \"dae66016-70ae-4239-9650-13a6c116d0c8\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") "
Jul 31 22:15:11.603127 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603129    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-envoy\") pod \"dae66016-70ae-4239-9650-13a6c116d0c8\" (UID: \"dae66016-70ae-4239-9650-13a6c116d0c8\") "
Jul 31 22:15:11.603498 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603040    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dae66016-70ae-4239-9650-13a6c116d0c8" (UID: "dae66016-70ae-4239-9650-13a6c116d0c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:15:11.603498 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603185    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "dae66016-70ae-4239-9650-13a6c116d0c8" (UID: "dae66016-70ae-4239-9650-13a6c116d0c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:15:11.603498 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603229    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dae66016-70ae-4239-9650-13a6c116d0c8" (UID: "dae66016-70ae-4239-9650-13a6c116d0c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:15:11.603498 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603271    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dae66016-70ae-4239-9650-13a6c116d0c8" (UID: "dae66016-70ae-4239-9650-13a6c116d0c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:15:11.603727 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603519    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-data\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:15:11.603727 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603540    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-credential-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:15:11.603727 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603553    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:15:11.603727 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603566    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-workload-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:15:11.603727 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.603563    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae66016-70ae-4239-9650-13a6c116d0c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dae66016-70ae-4239-9650-13a6c116d0c8" (UID: "dae66016-70ae-4239-9650-13a6c116d0c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:15:11.605241 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.605174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "dae66016-70ae-4239-9650-13a6c116d0c8" (UID: "dae66016-70ae-4239-9650-13a6c116d0c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:15:11.605380 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.605291    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dae66016-70ae-4239-9650-13a6c116d0c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dae66016-70ae-4239-9650-13a6c116d0c8" (UID: "dae66016-70ae-4239-9650-13a6c116d0c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 22:15:11.605380 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.605322    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-kube-api-access-slqxj" (OuterVolumeSpecName: "kube-api-access-slqxj") pod "dae66016-70ae-4239-9650-13a6c116d0c8" (UID: "dae66016-70ae-4239-9650-13a6c116d0c8"). InnerVolumeSpecName "kube-api-access-slqxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:15:11.605648 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.605632    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dae66016-70ae-4239-9650-13a6c116d0c8" (UID: "dae66016-70ae-4239-9650-13a6c116d0c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:15:11.704815 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.704771    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-istio-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:15:11.704815 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.704807    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae66016-70ae-4239-9650-13a6c116d0c8-istio-podinfo\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:15:11.704815 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.704817    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae66016-70ae-4239-9650-13a6c116d0c8-istiod-ca-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:15:11.704815 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.704825    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae66016-70ae-4239-9650-13a6c116d0c8-istio-envoy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:15:11.705100 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:11.704833    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slqxj\" (UniqueName: \"kubernetes.io/projected/dae66016-70ae-4239-9650-13a6c116d0c8-kube-api-access-slqxj\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:15:12.401060 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.401019    2561 generic.go:356] "Generic (PLEG): container finished" podID="dae66016-70ae-4239-9650-13a6c116d0c8" containerID="8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6" exitCode=0
Jul 31 22:15:12.401274 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.401076    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"
Jul 31 22:15:12.401274 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.401104    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" event={"ID":"dae66016-70ae-4239-9650-13a6c116d0c8","Type":"ContainerDied","Data":"8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6"}
Jul 31 22:15:12.401274 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.401189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt" event={"ID":"dae66016-70ae-4239-9650-13a6c116d0c8","Type":"ContainerDied","Data":"35d0f22646fea15184c55d6d93ffd774e4d4aeecdbdc97b5240081da6f8cb38b"}
Jul 31 22:15:12.401274 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.401216    2561 scope.go:117] "RemoveContainer" containerID="8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6"
Jul 31 22:15:12.410548 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.410522    2561 scope.go:117] "RemoveContainer" containerID="652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428"
Jul 31 22:15:12.419227 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.418006    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"]
Jul 31 22:15:12.422041 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.422018    2561 scope.go:117] "RemoveContainer" containerID="8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6"
Jul 31 22:15:12.422572 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:15:12.422471    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6\": container with ID starting with 8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6 not found: ID does not exist" containerID="8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6"
Jul 31 22:15:12.422572 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.422509    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6"} err="failed to get container status \"8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6\": rpc error: code = NotFound desc = could not find container \"8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6\": container with ID starting with 8d9bdf1c8c310f1a86fccbc3e8cbf6284380a7f0fba937eb4f53312eb4caf4b6 not found: ID does not exist"
Jul 31 22:15:12.422572 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.422537    2561 scope.go:117] "RemoveContainer" containerID="652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428"
Jul 31 22:15:12.423032 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:15:12.423006    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428\": container with ID starting with 652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428 not found: ID does not exist" containerID="652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428"
Jul 31 22:15:12.423156 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.423041    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428"} err="failed to get container status \"652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428\": rpc error: code = NotFound desc = could not find container \"652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428\": container with ID starting with 652b647ca9f6b9f4d639b51414493960df3b9edbdcc7f73cf8d790d532c17428 not found: ID does not exist"
Jul 31 22:15:12.424170 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:12.423851    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b0a39ff7-openshift-default-555c48dcfb-xg4kt"]
Jul 31 22:15:14.227874 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:15:14.227838    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" path="/var/lib/kubelet/pods/dae66016-70ae-4239-9650-13a6c116d0c8/volumes"
Jul 31 22:16:00.982763 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:00.982734    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:16:00.986742 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:00.986719    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:16:11.442048 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442008    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"]
Jul 31 22:16:11.442616 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442595    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy"
Jul 31 22:16:11.442667 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442622    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy"
Jul 31 22:16:11.442667 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442652    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3abb5d2-1386-408c-bce4-2b4e82aa4736" containerName="maas-api"
Jul 31 22:16:11.442667 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442661    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3abb5d2-1386-408c-bce4-2b4e82aa4736" containerName="maas-api"
Jul 31 22:16:11.442774 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442672    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy"
Jul 31 22:16:11.442774 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442681    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy"
Jul 31 22:16:11.442774 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442716    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bedbd439-0d07-4b19-83f4-7c093ece9b44" containerName="payload-processing"
Jul 31 22:16:11.442774 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442725    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bedbd439-0d07-4b19-83f4-7c093ece9b44" containerName="payload-processing"
Jul 31 22:16:11.442911 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442813    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3abb5d2-1386-408c-bce4-2b4e82aa4736" containerName="maas-api"
Jul 31 22:16:11.442911 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442826    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy"
Jul 31 22:16:11.442911 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.442836    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bedbd439-0d07-4b19-83f4-7c093ece9b44" containerName="payload-processing"
Jul 31 22:16:11.446270 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.446249    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.448309 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.448289    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 22:16:11.448454 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.448292    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0ac91e-openshift-default-dockercfg-chrvp\""
Jul 31 22:16:11.457938 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.457908    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"]
Jul 31 22:16:11.576026 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.575981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-envoy\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.576026 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.576025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-socket\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.576366 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.576046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-podinfo\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.576366 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.576189    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istiod-ca-cert\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.576366 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.576239    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-token\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.576366 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.576291    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-certs\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.576642 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.576384    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-credential-socket\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.576642 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.576403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-data\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.576642 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.576419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqx8\" (UniqueName: \"kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-kube-api-access-2cqx8\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.677704 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.677662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-token\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.677922 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.677722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-certs\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.677922 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.677782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-credential-socket\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.677922 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.677798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-data\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.677922 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.677821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqx8\" (UniqueName: \"kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-kube-api-access-2cqx8\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.677922 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.677876    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-envoy\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.677922 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.677906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-socket\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.678297 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.677929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-podinfo\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.678297 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.677962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istiod-ca-cert\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.678297 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.678268    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-certs\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.678459 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.678416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-credential-socket\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.678515 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.678473    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-data\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.678600 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.678577    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-socket\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.678806 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.678782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istiod-ca-cert\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.680311 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.680290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-envoy\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.680598 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.680577    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-podinfo\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.685648 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.685615    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqx8\" (UniqueName: \"kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-kube-api-access-2cqx8\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.685765 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.685670    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-token\") pod \"e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") " pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.757279 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.757176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:11.903032 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.903004    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"]
Jul 31 22:16:11.905734 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:16:11.905704    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55db13a2_0a54_4fb5_b9ef_e671c2678da0.slice/crio-f8223eaac5385c94512337fa6c72896cd23809da42f152a1edba471662b77022 WatchSource:0}: Error finding container f8223eaac5385c94512337fa6c72896cd23809da42f152a1edba471662b77022: Status 404 returned error can't find the container with id f8223eaac5385c94512337fa6c72896cd23809da42f152a1edba471662b77022
Jul 31 22:16:11.907837 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.907802    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:16:11.907904 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.907874    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:16:11.907942 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:11.907904    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:16:12.695824 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:12.695783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" event={"ID":"55db13a2-0a54-4fb5-b9ef-e671c2678da0","Type":"ContainerStarted","Data":"f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2"}
Jul 31 22:16:12.696236 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:12.695834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" event={"ID":"55db13a2-0a54-4fb5-b9ef-e671c2678da0","Type":"ContainerStarted","Data":"f8223eaac5385c94512337fa6c72896cd23809da42f152a1edba471662b77022"}
Jul 31 22:16:12.717066 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:12.717002    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" podStartSLOduration=1.7169832010000001 podStartE2EDuration="1.716983201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:16:12.714561042 +0000 UTC m=+2412.977309406" watchObservedRunningTime="2026-07-31 22:16:12.716983201 +0000 UTC m=+2412.979731639"
Jul 31 22:16:12.757542 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:12.757505    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:12.762734 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:12.762702    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:13.701032 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:13.701002    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:13.702218 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:13.702197    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:15.808608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:15.808567    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"]
Jul 31 22:16:15.809372 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:15.809349    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dae66016-70ae-4239-9650-13a6c116d0c8" containerName="istio-proxy"
Jul 31 22:16:15.812553 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:15.812532    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:15.814709 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:15.814679    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0ac91e-serving-cert\""
Jul 31 22:16:15.819230 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:15.819204    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"]
Jul 31 22:16:15.921246 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:15.921203    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:15.921436 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:15.921263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscz7\" (UniqueName: \"kubernetes.io/projected/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-kube-api-access-nscz7\") pod \"maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:15.921436 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:15.921286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-maas-api-tls\") pod \"maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:16.022317 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.022277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:16.022317 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.022323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscz7\" (UniqueName: \"kubernetes.io/projected/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-kube-api-access-nscz7\") pod \"maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:16.022543 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.022341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-maas-api-tls\") pod \"maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:16.023185 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.023118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:16.024931 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.024902    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-maas-api-tls\") pod \"maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:16.029891 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.029868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscz7\" (UniqueName: \"kubernetes.io/projected/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-kube-api-access-nscz7\") pod \"maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:16.127032 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.126934    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:16.271081 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.271023    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"]
Jul 31 22:16:16.273834 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:16:16.273791    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee93ba8b_af2e_4f3d_8962_e83803f5d5f8.slice/crio-5755fed325531295607dd7bcb64ee5ac3e096b5ae2078fa5ae0ef5199dab4837 WatchSource:0}: Error finding container 5755fed325531295607dd7bcb64ee5ac3e096b5ae2078fa5ae0ef5199dab4837: Status 404 returned error can't find the container with id 5755fed325531295607dd7bcb64ee5ac3e096b5ae2078fa5ae0ef5199dab4837
Jul 31 22:16:16.452092 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.452006    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"]
Jul 31 22:16:16.456879 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.456860    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:16:16.458975 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.458946    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0ac91e-dockercfg-k6hkn\""
Jul 31 22:16:16.459230 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.459214    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0ac91e\""
Jul 31 22:16:16.464651 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.464626    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"]
Jul 31 22:16:16.527645 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.527609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532f2cd2-eb2a-47d4-b222-0703b901f738-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd\" (UID: \"532f2cd2-eb2a-47d4-b222-0703b901f738\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:16:16.527645 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.527653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bl8\" (UniqueName: \"kubernetes.io/projected/532f2cd2-eb2a-47d4-b222-0703b901f738-kube-api-access-82bl8\") pod \"payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd\" (UID: \"532f2cd2-eb2a-47d4-b222-0703b901f738\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:16:16.628455 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.628412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532f2cd2-eb2a-47d4-b222-0703b901f738-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd\" (UID: \"532f2cd2-eb2a-47d4-b222-0703b901f738\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:16:16.628455 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.628454    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bl8\" (UniqueName: \"kubernetes.io/projected/532f2cd2-eb2a-47d4-b222-0703b901f738-kube-api-access-82bl8\") pod \"payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd\" (UID: \"532f2cd2-eb2a-47d4-b222-0703b901f738\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:16:16.629166 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.629117    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532f2cd2-eb2a-47d4-b222-0703b901f738-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd\" (UID: \"532f2cd2-eb2a-47d4-b222-0703b901f738\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:16:16.637516 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.637485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bl8\" (UniqueName: \"kubernetes.io/projected/532f2cd2-eb2a-47d4-b222-0703b901f738-kube-api-access-82bl8\") pod \"payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd\" (UID: \"532f2cd2-eb2a-47d4-b222-0703b901f738\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:16:16.716468 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.716375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj" event={"ID":"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8","Type":"ContainerStarted","Data":"5755fed325531295607dd7bcb64ee5ac3e096b5ae2078fa5ae0ef5199dab4837"}
Jul 31 22:16:16.769936 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.769908    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:16:16.917491 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:16.917460    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"]
Jul 31 22:16:16.919695 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:16:16.919663    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532f2cd2_eb2a_47d4_b222_0703b901f738.slice/crio-d150b1732b95d2657499cc60c85de4b47f7703fc4723a19dd34ea4a87235e79b WatchSource:0}: Error finding container d150b1732b95d2657499cc60c85de4b47f7703fc4723a19dd34ea4a87235e79b: Status 404 returned error can't find the container with id d150b1732b95d2657499cc60c85de4b47f7703fc4723a19dd34ea4a87235e79b
Jul 31 22:16:17.723446 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:17.723398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj" event={"ID":"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8","Type":"ContainerStarted","Data":"dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53"}
Jul 31 22:16:17.723664 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:17.723541    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:17.724889 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:17.724857    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd" event={"ID":"532f2cd2-eb2a-47d4-b222-0703b901f738","Type":"ContainerStarted","Data":"c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0"}
Jul 31 22:16:17.725038 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:17.724904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd" event={"ID":"532f2cd2-eb2a-47d4-b222-0703b901f738","Type":"ContainerStarted","Data":"d150b1732b95d2657499cc60c85de4b47f7703fc4723a19dd34ea4a87235e79b"}
Jul 31 22:16:17.725038 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:17.725002    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:16:17.738867 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:17.738811    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj" podStartSLOduration=2.256952921 podStartE2EDuration="2.738794683s" totalImagesPullingTime="481.841762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:16:17.737286113 +0000 UTC m=+2418.000034475" watchObservedRunningTime="2026-07-31 22:16:17.738794683 +0000 UTC m=+2418.001543047"
Jul 31 22:16:17.756090 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:17.756019    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd" podStartSLOduration=1.755996154 podStartE2EDuration="1.755996154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:16:17.755502662 +0000 UTC m=+2418.018251026" watchObservedRunningTime="2026-07-31 22:16:17.755996154 +0000 UTC m=+2418.018744518"
Jul 31 22:16:23.735353 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:23.735269    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:16:28.702307 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:28.702267    2561 patch_prober.go:28] interesting pod/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 31 22:16:28.702726 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:28.702333    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 31 22:16:28.731806 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:28.731757    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd" podUID="532f2cd2-eb2a-47d4-b222-0703b901f738" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.61:9004: connect: connection refused"
Jul 31 22:16:28.780745 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:28.780706    2561 generic.go:356] "Generic (PLEG): container finished" podID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerID="f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2" exitCode=0
Jul 31 22:16:28.780949 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:28.780786    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" event={"ID":"55db13a2-0a54-4fb5-b9ef-e671c2678da0","Type":"ContainerDied","Data":"f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2"}
Jul 31 22:16:28.781242 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:28.781228    2561 scope.go:117] "RemoveContainer" containerID="f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2"
Jul 31 22:16:28.782278 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:28.782250    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:16:28.782362 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:28.782320    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:16:28.782362 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:28.782353    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:16:29.787869 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:29.787822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" event={"ID":"55db13a2-0a54-4fb5-b9ef-e671c2678da0","Type":"ContainerStarted","Data":"4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2"}
Jul 31 22:16:30.757922 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:30.757886    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:30.758239 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:30.758215    2561 patch_prober.go:28] interesting pod/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 31 22:16:30.758302 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:30.758269    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 31 22:16:31.757629 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:31.757589    2561 patch_prober.go:28] interesting pod/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 31 22:16:31.758270 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:31.757650    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 31 22:16:32.758540 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:32.758502    2561 patch_prober.go:28] interesting pod/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 31 22:16:32.758982 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:32.758562    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 31 22:16:33.758453 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:33.758413    2561 patch_prober.go:28] interesting pod/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 31 22:16:33.758655 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:33.758487    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 31 22:16:34.761668 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:34.761635    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:34.762090 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:34.761914    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:34.762674 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:34.762655    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:16:38.730678 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:38.730620    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:16:44.740051 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.740012    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"]
Jul 31 22:16:44.743736 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.743715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:44.747817 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.747790    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d0110c-serving-cert\""
Jul 31 22:16:44.759262 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.759232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"]
Jul 31 22:16:44.800655 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.800617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp9lk\" (UniqueName: \"kubernetes.io/projected/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-kube-api-access-bp9lk\") pod \"maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:44.800837 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.800684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-maas-api-tls\") pod \"maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:44.800837 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.800740    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:44.902308 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.902269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp9lk\" (UniqueName: \"kubernetes.io/projected/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-kube-api-access-bp9lk\") pod \"maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:44.902495 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.902326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-maas-api-tls\") pod \"maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:44.902495 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.902362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:44.902495 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:16:44.902471    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-serving-cert: secret "maas-api-e2e-inf-b-d0110c-serving-cert" not found
Jul 31 22:16:44.902610 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:16:44.902553    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-maas-api-tls podName:1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43 nodeName:}" failed. No retries permitted until 2026-07-31 22:16:45.402532831 +0000 UTC m=+2445.665281180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-maas-api-tls") pod "maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9" (UID: "1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43") : secret "maas-api-e2e-inf-b-d0110c-serving-cert" not found
Jul 31 22:16:44.903029 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.903010    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:44.912249 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:44.912218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp9lk\" (UniqueName: \"kubernetes.io/projected/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-kube-api-access-bp9lk\") pod \"maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:45.369383 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.369346    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"]
Jul 31 22:16:45.373123 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.373105    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:16:45.375004 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.374978    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d0110c-dockercfg-j6p4n\""
Jul 31 22:16:45.375004 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.374998    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d0110c\""
Jul 31 22:16:45.383575 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.383540    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"]
Jul 31 22:16:45.390667 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.390638    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"]
Jul 31 22:16:45.395413 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.394954    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:16:45.408472 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.408434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvrds\" (UniqueName: \"kubernetes.io/projected/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-kube-api-access-fvrds\") pod \"payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5\" (UID: \"41e58ef2-f61b-4ea0-8e75-fd50fae10b66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:16:45.408644 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.408541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-maas-api-tls\") pod \"maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:45.408644 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.408581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5\" (UID: \"41e58ef2-f61b-4ea0-8e75-fd50fae10b66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:16:45.409864 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.409842    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"]
Jul 31 22:16:45.411768 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.411733    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-maas-api-tls\") pod \"maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:45.510063 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.510019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5tp2\" (UniqueName: \"kubernetes.io/projected/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-kube-api-access-t5tp2\") pod \"payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn\" (UID: \"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:16:45.510306 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.510073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5\" (UID: \"41e58ef2-f61b-4ea0-8e75-fd50fae10b66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:16:45.510306 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.510246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvrds\" (UniqueName: \"kubernetes.io/projected/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-kube-api-access-fvrds\") pod \"payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5\" (UID: \"41e58ef2-f61b-4ea0-8e75-fd50fae10b66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:16:45.510306 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.510287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn\" (UID: \"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:16:45.510867 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.510849    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5\" (UID: \"41e58ef2-f61b-4ea0-8e75-fd50fae10b66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:16:45.518501 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.518473    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvrds\" (UniqueName: \"kubernetes.io/projected/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-kube-api-access-fvrds\") pod \"payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5\" (UID: \"41e58ef2-f61b-4ea0-8e75-fd50fae10b66\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:16:45.611528 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.611494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn\" (UID: \"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:16:45.611748 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.611555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5tp2\" (UniqueName: \"kubernetes.io/projected/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-kube-api-access-t5tp2\") pod \"payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn\" (UID: \"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:16:45.612326 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.612298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn\" (UID: \"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:16:45.620951 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.620881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5tp2\" (UniqueName: \"kubernetes.io/projected/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-kube-api-access-t5tp2\") pod \"payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn\" (UID: \"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:16:45.656208 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.656170    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:45.685525 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.685498    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:16:45.706431 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.706396    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:16:45.831172 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.831058    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"]
Jul 31 22:16:45.834417 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:16:45.834371    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b80d54a_d0fa_46aa_bcbd_bd4bc820bf43.slice/crio-414963e4d9965557bb9563cdb2383003fcc7bad4c4ab1fc0701fd6b5ed52772f WatchSource:0}: Error finding container 414963e4d9965557bb9563cdb2383003fcc7bad4c4ab1fc0701fd6b5ed52772f: Status 404 returned error can't find the container with id 414963e4d9965557bb9563cdb2383003fcc7bad4c4ab1fc0701fd6b5ed52772f
Jul 31 22:16:45.861936 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.861791    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"]
Jul 31 22:16:45.864851 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:16:45.864824    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e58ef2_f61b_4ea0_8e75_fd50fae10b66.slice/crio-ae45b8a9f78ac622ff709ef480b3413e668cefcc50f9cae6ac07383c93c67b73 WatchSource:0}: Error finding container ae45b8a9f78ac622ff709ef480b3413e668cefcc50f9cae6ac07383c93c67b73: Status 404 returned error can't find the container with id ae45b8a9f78ac622ff709ef480b3413e668cefcc50f9cae6ac07383c93c67b73
Jul 31 22:16:45.885174 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.885112    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9" event={"ID":"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43","Type":"ContainerStarted","Data":"414963e4d9965557bb9563cdb2383003fcc7bad4c4ab1fc0701fd6b5ed52772f"}
Jul 31 22:16:45.886403 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.886373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5" event={"ID":"41e58ef2-f61b-4ea0-8e75-fd50fae10b66","Type":"ContainerStarted","Data":"ae45b8a9f78ac622ff709ef480b3413e668cefcc50f9cae6ac07383c93c67b73"}
Jul 31 22:16:45.895607 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:45.895582    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"]
Jul 31 22:16:45.907457 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:16:45.907426    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1839cf_9c63_4f84_a98c_66fcf7b2e2f2.slice/crio-da43fdd71d6e547585e23d9ee85b66a089ec76cba89f2c22fdbf8a32dda02677 WatchSource:0}: Error finding container da43fdd71d6e547585e23d9ee85b66a089ec76cba89f2c22fdbf8a32dda02677: Status 404 returned error can't find the container with id da43fdd71d6e547585e23d9ee85b66a089ec76cba89f2c22fdbf8a32dda02677
Jul 31 22:16:46.892969 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:46.892930    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9" event={"ID":"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43","Type":"ContainerStarted","Data":"ada82bae991a1dc7e9782735ed2d7ea75fc17db72b91cf0c47d908835bca99ee"}
Jul 31 22:16:46.893487 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:46.893129    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:46.894709 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:46.894679    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5" event={"ID":"41e58ef2-f61b-4ea0-8e75-fd50fae10b66","Type":"ContainerStarted","Data":"85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51"}
Jul 31 22:16:46.894835 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:46.894736    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:16:46.896335 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:46.896309    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn" event={"ID":"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2","Type":"ContainerStarted","Data":"c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071"}
Jul 31 22:16:46.896433 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:46.896357    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn" event={"ID":"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2","Type":"ContainerStarted","Data":"da43fdd71d6e547585e23d9ee85b66a089ec76cba89f2c22fdbf8a32dda02677"}
Jul 31 22:16:46.896433 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:46.896386    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:16:46.908762 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:46.908706    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9" podStartSLOduration=2.465747018 podStartE2EDuration="2.908689437s" totalImagesPullingTime="442.942419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:16:46.907237955 +0000 UTC m=+2447.169986319" watchObservedRunningTime="2026-07-31 22:16:46.908689437 +0000 UTC m=+2447.171437800"
Jul 31 22:16:46.924044 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:46.923985    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn" podStartSLOduration=1.92396758 podStartE2EDuration="1.92396758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:16:46.922531995 +0000 UTC m=+2447.185280360" watchObservedRunningTime="2026-07-31 22:16:46.92396758 +0000 UTC m=+2447.186715943"
Jul 31 22:16:46.939726 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:46.939664    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5" podStartSLOduration=1.93964654 podStartE2EDuration="1.93964654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:16:46.93727085 +0000 UTC m=+2447.200019216" watchObservedRunningTime="2026-07-31 22:16:46.93964654 +0000 UTC m=+2447.202394904"
Jul 31 22:16:52.907324 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:52.907290    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:16:57.903056 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:57.903022    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:16:57.903466 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:16:57.903097    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:18:40.395839 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.395804    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"]
Jul 31 22:18:40.396768 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.396056    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj" podUID="ee93ba8b-af2e-4f3d-8962-e83803f5d5f8" containerName="maas-api" containerID="cri-o://dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53" gracePeriod=30
Jul 31 22:18:40.451811 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.451158    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"]
Jul 31 22:18:40.452270 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.452000    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd" podUID="532f2cd2-eb2a-47d4-b222-0703b901f738" containerName="payload-pre-processing" containerID="cri-o://c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0" gracePeriod=30
Jul 31 22:18:40.710097 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.710069    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:18:40.745412 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.745381    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:18:40.805997 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.805959    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82bl8\" (UniqueName: \"kubernetes.io/projected/532f2cd2-eb2a-47d4-b222-0703b901f738-kube-api-access-82bl8\") pod \"532f2cd2-eb2a-47d4-b222-0703b901f738\" (UID: \"532f2cd2-eb2a-47d4-b222-0703b901f738\") "
Jul 31 22:18:40.806211 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.806014    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-openshift-service-ca\") pod \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") "
Jul 31 22:18:40.806211 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.806160    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-maas-api-tls\") pod \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") "
Jul 31 22:18:40.806338 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.806226    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nscz7\" (UniqueName: \"kubernetes.io/projected/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-kube-api-access-nscz7\") pod \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\" (UID: \"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8\") "
Jul 31 22:18:40.806338 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.806283    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532f2cd2-eb2a-47d4-b222-0703b901f738-plugins-config-volume\") pod \"532f2cd2-eb2a-47d4-b222-0703b901f738\" (UID: \"532f2cd2-eb2a-47d4-b222-0703b901f738\") "
Jul 31 22:18:40.806441 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.806339    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee93ba8b-af2e-4f3d-8962-e83803f5d5f8" (UID: "ee93ba8b-af2e-4f3d-8962-e83803f5d5f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:18:40.806679 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.806654    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532f2cd2-eb2a-47d4-b222-0703b901f738-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "532f2cd2-eb2a-47d4-b222-0703b901f738" (UID: "532f2cd2-eb2a-47d4-b222-0703b901f738"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:18:40.806679 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.806673    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-openshift-service-ca\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:40.808318 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.808294    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee93ba8b-af2e-4f3d-8962-e83803f5d5f8" (UID: "ee93ba8b-af2e-4f3d-8962-e83803f5d5f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:18:40.808318 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.808314    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532f2cd2-eb2a-47d4-b222-0703b901f738-kube-api-access-82bl8" (OuterVolumeSpecName: "kube-api-access-82bl8") pod "532f2cd2-eb2a-47d4-b222-0703b901f738" (UID: "532f2cd2-eb2a-47d4-b222-0703b901f738"). InnerVolumeSpecName "kube-api-access-82bl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:18:40.808494 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.808357    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-kube-api-access-nscz7" (OuterVolumeSpecName: "kube-api-access-nscz7") pod "ee93ba8b-af2e-4f3d-8962-e83803f5d5f8" (UID: "ee93ba8b-af2e-4f3d-8962-e83803f5d5f8"). InnerVolumeSpecName "kube-api-access-nscz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:18:40.907145 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.907104    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nscz7\" (UniqueName: \"kubernetes.io/projected/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-kube-api-access-nscz7\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:40.907145 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.907153    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532f2cd2-eb2a-47d4-b222-0703b901f738-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:40.907397 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.907167    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82bl8\" (UniqueName: \"kubernetes.io/projected/532f2cd2-eb2a-47d4-b222-0703b901f738-kube-api-access-82bl8\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:40.907397 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:40.907181    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8-maas-api-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:41.468875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.468834    2561 generic.go:356] "Generic (PLEG): container finished" podID="532f2cd2-eb2a-47d4-b222-0703b901f738" containerID="c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0" exitCode=0
Jul 31 22:18:41.469391 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.468881    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"
Jul 31 22:18:41.469391 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.468915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd" event={"ID":"532f2cd2-eb2a-47d4-b222-0703b901f738","Type":"ContainerDied","Data":"c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0"}
Jul 31 22:18:41.469391 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.468968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd" event={"ID":"532f2cd2-eb2a-47d4-b222-0703b901f738","Type":"ContainerDied","Data":"d150b1732b95d2657499cc60c85de4b47f7703fc4723a19dd34ea4a87235e79b"}
Jul 31 22:18:41.469391 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.468990    2561 scope.go:117] "RemoveContainer" containerID="c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0"
Jul 31 22:18:41.470423 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.470400    2561 generic.go:356] "Generic (PLEG): container finished" podID="ee93ba8b-af2e-4f3d-8962-e83803f5d5f8" containerID="dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53" exitCode=0
Jul 31 22:18:41.470489 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.470437    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"
Jul 31 22:18:41.470537 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.470482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj" event={"ID":"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8","Type":"ContainerDied","Data":"dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53"}
Jul 31 22:18:41.470577 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.470536    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj" event={"ID":"ee93ba8b-af2e-4f3d-8962-e83803f5d5f8","Type":"ContainerDied","Data":"5755fed325531295607dd7bcb64ee5ac3e096b5ae2078fa5ae0ef5199dab4837"}
Jul 31 22:18:41.480634 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.480592    2561 scope.go:117] "RemoveContainer" containerID="c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0"
Jul 31 22:18:41.481419 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:18:41.481070    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0\": container with ID starting with c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0 not found: ID does not exist" containerID="c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0"
Jul 31 22:18:41.481419 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.481099    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0"} err="failed to get container status \"c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0\": rpc error: code = NotFound desc = could not find container \"c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0\": container with ID starting with c64f437e7a1e451bd9268759ebc93ebee8ca4db0dc543a08549ae7db6e57d2a0 not found: ID does not exist"
Jul 31 22:18:41.481419 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.481119    2561 scope.go:117] "RemoveContainer" containerID="dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53"
Jul 31 22:18:41.490519 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.490498    2561 scope.go:117] "RemoveContainer" containerID="dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53"
Jul 31 22:18:41.490851 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:18:41.490830    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53\": container with ID starting with dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53 not found: ID does not exist" containerID="dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53"
Jul 31 22:18:41.490913 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.490860    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53"} err="failed to get container status \"dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53\": rpc error: code = NotFound desc = could not find container \"dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53\": container with ID starting with dfa4a6bb123e4afe4e0194ebcefe1a2cc5aa0a3b36cff0cfa53a11305e81eb53 not found: ID does not exist"
Jul 31 22:18:41.496653 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.496624    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"]
Jul 31 22:18:41.500034 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.500009    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0ac91e-67556955c4-pv9mj"]
Jul 31 22:18:41.509520 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.509485    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"]
Jul 31 22:18:41.511513 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:41.511493    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0ac91e-66655cf4d8-lhjfd"]
Jul 31 22:18:42.227754 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:42.227718    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="532f2cd2-eb2a-47d4-b222-0703b901f738" path="/var/lib/kubelet/pods/532f2cd2-eb2a-47d4-b222-0703b901f738/volumes"
Jul 31 22:18:42.228117 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:42.228103    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee93ba8b-af2e-4f3d-8962-e83803f5d5f8" path="/var/lib/kubelet/pods/ee93ba8b-af2e-4f3d-8962-e83803f5d5f8/volumes"
Jul 31 22:18:47.123883 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:47.123829    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"]
Jul 31 22:18:47.124420 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:47.124097    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy" containerID="cri-o://4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2" gracePeriod=30
Jul 31 22:18:49.764594 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:49.764544    2561 patch_prober.go:28] interesting pod/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 22:18:49.764984 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:49.764611    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 22:18:52.383683 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.383655    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:18:52.418802 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.418694    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-credential-socket\") pod \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") "
Jul 31 22:18:52.418802 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.418736    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-podinfo\") pod \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") "
Jul 31 22:18:52.419058 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.418810    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-token\") pod \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") "
Jul 31 22:18:52.419058 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.418831    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-envoy\") pod \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") "
Jul 31 22:18:52.419058 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.418863    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cqx8\" (UniqueName: \"kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-kube-api-access-2cqx8\") pod \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") "
Jul 31 22:18:52.419058 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.418918    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istiod-ca-cert\") pod \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") "
Jul 31 22:18:52.419058 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.418943    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-data\") pod \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") "
Jul 31 22:18:52.419058 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.418971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-socket\") pod \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") "
Jul 31 22:18:52.419058 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.418993    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-certs\") pod \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\" (UID: \"55db13a2-0a54-4fb5-b9ef-e671c2678da0\") "
Jul 31 22:18:52.419058 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.419006    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "55db13a2-0a54-4fb5-b9ef-e671c2678da0" (UID: "55db13a2-0a54-4fb5-b9ef-e671c2678da0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:18:52.419528 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.419326    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "55db13a2-0a54-4fb5-b9ef-e671c2678da0" (UID: "55db13a2-0a54-4fb5-b9ef-e671c2678da0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:18:52.419528 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.419377    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-credential-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:52.419646 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.419565    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-data" (OuterVolumeSpecName: "istio-data") pod "55db13a2-0a54-4fb5-b9ef-e671c2678da0" (UID: "55db13a2-0a54-4fb5-b9ef-e671c2678da0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:18:52.419646 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.419576    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "55db13a2-0a54-4fb5-b9ef-e671c2678da0" (UID: "55db13a2-0a54-4fb5-b9ef-e671c2678da0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:18:52.419761 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.419723    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "55db13a2-0a54-4fb5-b9ef-e671c2678da0" (UID: "55db13a2-0a54-4fb5-b9ef-e671c2678da0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:18:52.421358 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.421316    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-token" (OuterVolumeSpecName: "istio-token") pod "55db13a2-0a54-4fb5-b9ef-e671c2678da0" (UID: "55db13a2-0a54-4fb5-b9ef-e671c2678da0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:18:52.421478 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.421419    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "55db13a2-0a54-4fb5-b9ef-e671c2678da0" (UID: "55db13a2-0a54-4fb5-b9ef-e671c2678da0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 22:18:52.421590 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.421571    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "55db13a2-0a54-4fb5-b9ef-e671c2678da0" (UID: "55db13a2-0a54-4fb5-b9ef-e671c2678da0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:18:52.421629 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.421604    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-kube-api-access-2cqx8" (OuterVolumeSpecName: "kube-api-access-2cqx8") pod "55db13a2-0a54-4fb5-b9ef-e671c2678da0" (UID: "55db13a2-0a54-4fb5-b9ef-e671c2678da0"). InnerVolumeSpecName "kube-api-access-2cqx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:18:52.520509 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.520466    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:52.520509 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.520500    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-envoy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:52.520509 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.520510    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqx8\" (UniqueName: \"kubernetes.io/projected/55db13a2-0a54-4fb5-b9ef-e671c2678da0-kube-api-access-2cqx8\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:52.520509 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.520519    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istiod-ca-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:52.520777 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.520527    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-data\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:52.520777 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.520535    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:52.520777 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.520543    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55db13a2-0a54-4fb5-b9ef-e671c2678da0-workload-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:52.520777 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.520551    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55db13a2-0a54-4fb5-b9ef-e671c2678da0-istio-podinfo\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:52.529165 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.529110    2561 generic.go:356] "Generic (PLEG): container finished" podID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerID="4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2" exitCode=0
Jul 31 22:18:52.529329 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.529174    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"
Jul 31 22:18:52.529329 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.529193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" event={"ID":"55db13a2-0a54-4fb5-b9ef-e671c2678da0","Type":"ContainerDied","Data":"4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2"}
Jul 31 22:18:52.529329 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.529249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp" event={"ID":"55db13a2-0a54-4fb5-b9ef-e671c2678da0","Type":"ContainerDied","Data":"f8223eaac5385c94512337fa6c72896cd23809da42f152a1edba471662b77022"}
Jul 31 22:18:52.529329 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.529273    2561 scope.go:117] "RemoveContainer" containerID="4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2"
Jul 31 22:18:52.538907 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.538889    2561 scope.go:117] "RemoveContainer" containerID="f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2"
Jul 31 22:18:52.547416 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.547396    2561 scope.go:117] "RemoveContainer" containerID="4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2"
Jul 31 22:18:52.547712 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:18:52.547692    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2\": container with ID starting with 4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2 not found: ID does not exist" containerID="4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2"
Jul 31 22:18:52.547787 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.547726    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2"} err="failed to get container status \"4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2\": rpc error: code = NotFound desc = could not find container \"4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2\": container with ID starting with 4ad899be75573d7db3645865eedb3e863d93d85bc4df4320df87ee1a832f31f2 not found: ID does not exist"
Jul 31 22:18:52.547787 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.547754    2561 scope.go:117] "RemoveContainer" containerID="f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2"
Jul 31 22:18:52.548018 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:18:52.548001    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2\": container with ID starting with f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2 not found: ID does not exist" containerID="f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2"
Jul 31 22:18:52.548057 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.548026    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2"} err="failed to get container status \"f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2\": rpc error: code = NotFound desc = could not find container \"f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2\": container with ID starting with f395b89b42cc453ea3c5560a3ced57b2366709d22dfa1d9f7ca3b1f89b8628f2 not found: ID does not exist"
Jul 31 22:18:52.550744 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.550722    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"]
Jul 31 22:18:52.555176 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:52.555153    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0ac91e-openshift-default-5f65cc74dd-gtdbp"]
Jul 31 22:18:54.234334 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:54.234295    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" path="/var/lib/kubelet/pods/55db13a2-0a54-4fb5-b9ef-e671c2678da0/volumes"
Jul 31 22:18:58.132398 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.132359    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"]
Jul 31 22:18:58.133183 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.133124    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9" podUID="1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43" containerName="maas-api" containerID="cri-o://ada82bae991a1dc7e9782735ed2d7ea75fc17db72b91cf0c47d908835bca99ee" gracePeriod=30
Jul 31 22:18:58.193090 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.193052    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"]
Jul 31 22:18:58.193382 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.193348    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn" podUID="5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2" containerName="payload-processing" containerID="cri-o://c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071" gracePeriod=30
Jul 31 22:18:58.201442 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.201407    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"]
Jul 31 22:18:58.201681 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.201652    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5" podUID="41e58ef2-f61b-4ea0-8e75-fd50fae10b66" containerName="payload-pre-processing" containerID="cri-o://85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51" gracePeriod=30
Jul 31 22:18:58.494544 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.494516    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:18:58.497484 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.497457    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:18:58.559585 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.559546    2561 generic.go:356] "Generic (PLEG): container finished" podID="5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2" containerID="c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071" exitCode=0
Jul 31 22:18:58.559767 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.559613    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"
Jul 31 22:18:58.559767 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.559640    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn" event={"ID":"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2","Type":"ContainerDied","Data":"c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071"}
Jul 31 22:18:58.559767 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.559698    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn" event={"ID":"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2","Type":"ContainerDied","Data":"da43fdd71d6e547585e23d9ee85b66a089ec76cba89f2c22fdbf8a32dda02677"}
Jul 31 22:18:58.559767 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.559731    2561 scope.go:117] "RemoveContainer" containerID="c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071"
Jul 31 22:18:58.561589 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.561566    2561 generic.go:356] "Generic (PLEG): container finished" podID="1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43" containerID="ada82bae991a1dc7e9782735ed2d7ea75fc17db72b91cf0c47d908835bca99ee" exitCode=0
Jul 31 22:18:58.561700 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.561641    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9" event={"ID":"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43","Type":"ContainerDied","Data":"ada82bae991a1dc7e9782735ed2d7ea75fc17db72b91cf0c47d908835bca99ee"}
Jul 31 22:18:58.563267 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.563241    2561 generic.go:356] "Generic (PLEG): container finished" podID="41e58ef2-f61b-4ea0-8e75-fd50fae10b66" containerID="85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51" exitCode=0
Jul 31 22:18:58.563395 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.563290    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5" event={"ID":"41e58ef2-f61b-4ea0-8e75-fd50fae10b66","Type":"ContainerDied","Data":"85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51"}
Jul 31 22:18:58.563395 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.563305    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"
Jul 31 22:18:58.563395 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.563330    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5" event={"ID":"41e58ef2-f61b-4ea0-8e75-fd50fae10b66","Type":"ContainerDied","Data":"ae45b8a9f78ac622ff709ef480b3413e668cefcc50f9cae6ac07383c93c67b73"}
Jul 31 22:18:58.569091 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.569074    2561 scope.go:117] "RemoveContainer" containerID="c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071"
Jul 31 22:18:58.569407 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:18:58.569388    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071\": container with ID starting with c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071 not found: ID does not exist" containerID="c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071"
Jul 31 22:18:58.569511 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.569419    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071"} err="failed to get container status \"c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071\": rpc error: code = NotFound desc = could not find container \"c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071\": container with ID starting with c4229af790b32746d40cd2eaeb340efa92d4857eea8ea79a95c123b339adf071 not found: ID does not exist"
Jul 31 22:18:58.569511 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.569449    2561 scope.go:117] "RemoveContainer" containerID="85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51"
Jul 31 22:18:58.577812 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.577790    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5tp2\" (UniqueName: \"kubernetes.io/projected/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-kube-api-access-t5tp2\") pod \"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2\" (UID: \"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2\") "
Jul 31 22:18:58.577934 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.577821    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-plugins-config-volume\") pod \"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2\" (UID: \"5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2\") "
Jul 31 22:18:58.577934 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.577847    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvrds\" (UniqueName: \"kubernetes.io/projected/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-kube-api-access-fvrds\") pod \"41e58ef2-f61b-4ea0-8e75-fd50fae10b66\" (UID: \"41e58ef2-f61b-4ea0-8e75-fd50fae10b66\") "
Jul 31 22:18:58.577934 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.577862    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-plugins-config-volume\") pod \"41e58ef2-f61b-4ea0-8e75-fd50fae10b66\" (UID: \"41e58ef2-f61b-4ea0-8e75-fd50fae10b66\") "
Jul 31 22:18:58.578323 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.578263    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2" (UID: "5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:18:58.578418 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.578357    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41e58ef2-f61b-4ea0-8e75-fd50fae10b66" (UID: "41e58ef2-f61b-4ea0-8e75-fd50fae10b66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:18:58.580016 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.579994    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-kube-api-access-fvrds" (OuterVolumeSpecName: "kube-api-access-fvrds") pod "41e58ef2-f61b-4ea0-8e75-fd50fae10b66" (UID: "41e58ef2-f61b-4ea0-8e75-fd50fae10b66"). InnerVolumeSpecName "kube-api-access-fvrds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:18:58.580317 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.580294    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-kube-api-access-t5tp2" (OuterVolumeSpecName: "kube-api-access-t5tp2") pod "5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2" (UID: "5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2"). InnerVolumeSpecName "kube-api-access-t5tp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:18:58.584589 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.584573    2561 scope.go:117] "RemoveContainer" containerID="85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51"
Jul 31 22:18:58.584841 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:18:58.584818    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51\": container with ID starting with 85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51 not found: ID does not exist" containerID="85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51"
Jul 31 22:18:58.584928 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.584849    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51"} err="failed to get container status \"85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51\": rpc error: code = NotFound desc = could not find container \"85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51\": container with ID starting with 85e7522a0c2ca0d8f38248dfbd0fd0387ecd6e0d55293c1ca6bed12e91d6ae51 not found: ID does not exist"
Jul 31 22:18:58.588455 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.588435    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:18:58.679062 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.678965    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-maas-api-tls\") pod \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") "
Jul 31 22:18:58.679062 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.679040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-openshift-service-ca\") pod \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") "
Jul 31 22:18:58.679062 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.679064    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp9lk\" (UniqueName: \"kubernetes.io/projected/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-kube-api-access-bp9lk\") pod \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\" (UID: \"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43\") "
Jul 31 22:18:58.679372 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.679351    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5tp2\" (UniqueName: \"kubernetes.io/projected/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-kube-api-access-t5tp2\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:58.679413 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.679380    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:58.679413 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.679395    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvrds\" (UniqueName: \"kubernetes.io/projected/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-kube-api-access-fvrds\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:58.679482 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.679410    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e58ef2-f61b-4ea0-8e75-fd50fae10b66-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:58.679482 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.679416    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43" (UID: "1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:18:58.681180 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.681157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-kube-api-access-bp9lk" (OuterVolumeSpecName: "kube-api-access-bp9lk") pod "1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43" (UID: "1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43"). InnerVolumeSpecName "kube-api-access-bp9lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:18:58.681235 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.681162    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43" (UID: "1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:18:58.780549 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.780510    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-openshift-service-ca\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:58.780549 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.780542    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp9lk\" (UniqueName: \"kubernetes.io/projected/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-kube-api-access-bp9lk\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:58.780549 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.780553    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43-maas-api-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:18:58.880643 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.880611    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"]
Jul 31 22:18:58.884492 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.884466    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0110c-7bb7779fc8-w5xmn"]
Jul 31 22:18:58.894276 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.894244    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"]
Jul 31 22:18:58.896518 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:58.896494    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0110c-d48d4fcb8-j7rx5"]
Jul 31 22:18:59.571209 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:59.571173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9" event={"ID":"1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43","Type":"ContainerDied","Data":"414963e4d9965557bb9563cdb2383003fcc7bad4c4ab1fc0701fd6b5ed52772f"}
Jul 31 22:18:59.571209 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:59.571211    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"
Jul 31 22:18:59.571734 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:59.571235    2561 scope.go:117] "RemoveContainer" containerID="ada82bae991a1dc7e9782735ed2d7ea75fc17db72b91cf0c47d908835bca99ee"
Jul 31 22:18:59.592422 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:59.592391    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"]
Jul 31 22:18:59.595576 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:18:59.595549    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0110c-7859877cd7-vkwc9"]
Jul 31 22:19:00.230688 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:00.230654    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43" path="/var/lib/kubelet/pods/1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43/volumes"
Jul 31 22:19:00.231026 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:00.231014    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e58ef2-f61b-4ea0-8e75-fd50fae10b66" path="/var/lib/kubelet/pods/41e58ef2-f61b-4ea0-8e75-fd50fae10b66/volumes"
Jul 31 22:19:00.231363 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:00.231351    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2" path="/var/lib/kubelet/pods/5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2/volumes"
Jul 31 22:19:05.610507 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.610473    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"]
Jul 31 22:19:05.611019 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.610998    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy"
Jul 31 22:19:05.611110 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611023    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy"
Jul 31 22:19:05.611110 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611046    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2" containerName="payload-processing"
Jul 31 22:19:05.611110 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611055    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2" containerName="payload-processing"
Jul 31 22:19:05.611110 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611067    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="532f2cd2-eb2a-47d4-b222-0703b901f738" containerName="payload-pre-processing"
Jul 31 22:19:05.611110 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611077    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="532f2cd2-eb2a-47d4-b222-0703b901f738" containerName="payload-pre-processing"
Jul 31 22:19:05.611110 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611105    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee93ba8b-af2e-4f3d-8962-e83803f5d5f8" containerName="maas-api"
Jul 31 22:19:05.611428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611114    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee93ba8b-af2e-4f3d-8962-e83803f5d5f8" containerName="maas-api"
Jul 31 22:19:05.611428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611148    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e58ef2-f61b-4ea0-8e75-fd50fae10b66" containerName="payload-pre-processing"
Jul 31 22:19:05.611428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611158    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e58ef2-f61b-4ea0-8e75-fd50fae10b66" containerName="payload-pre-processing"
Jul 31 22:19:05.611428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611170    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43" containerName="maas-api"
Jul 31 22:19:05.611428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611178    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43" containerName="maas-api"
Jul 31 22:19:05.611428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611276    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c1839cf-9c63-4f84-a98c-66fcf7b2e2f2" containerName="payload-processing"
Jul 31 22:19:05.611428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611291    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy"
Jul 31 22:19:05.611428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611302    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="532f2cd2-eb2a-47d4-b222-0703b901f738" containerName="payload-pre-processing"
Jul 31 22:19:05.611428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611313    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b80d54a-d0fa-46aa-bcbd-bd4bc820bf43" containerName="maas-api"
Jul 31 22:19:05.611428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611324    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e58ef2-f61b-4ea0-8e75-fd50fae10b66" containerName="payload-pre-processing"
Jul 31 22:19:05.611428 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.611334    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee93ba8b-af2e-4f3d-8962-e83803f5d5f8" containerName="maas-api"
Jul 31 22:19:05.616379 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.616355    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.618643 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.618619    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 22:19:05.618765 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.618619    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ef8f59-openshift-default-dockercfg-7cfhk\""
Jul 31 22:19:05.625820 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.625784    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"]
Jul 31 22:19:05.742782 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.742746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-envoy\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.743012 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.742789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcjpb\" (UniqueName: \"kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-kube-api-access-lcjpb\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.743012 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.742819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-data\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.743012 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.742856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-credential-socket\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.743012 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.742907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-certs\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.743012 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.742924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-token\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.743012 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.742962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-podinfo\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.743271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.743060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-socket\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.743271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.743089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92893c4d-43d6-42e4-a244-d7247e9acc44-istiod-ca-cert\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844021 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.843980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-credential-socket\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-certs\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-token\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-podinfo\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-socket\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92893c4d-43d6-42e4-a244-d7247e9acc44-istiod-ca-cert\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-envoy\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcjpb\" (UniqueName: \"kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-kube-api-access-lcjpb\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844632 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-data\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844632 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-credential-socket\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844632 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844512    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-certs\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844632 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844577    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-socket\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.844822 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.844692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-data\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.845093 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.845074    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92893c4d-43d6-42e4-a244-d7247e9acc44-istiod-ca-cert\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.846535 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.846514    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-envoy\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.846673 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.846656    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-podinfo\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.852888 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.852862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-token\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.853023 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.852995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcjpb\" (UniqueName: \"kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-kube-api-access-lcjpb\") pod \"e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") " pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:05.929199 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:05.929098    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:06.070498 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:06.070466    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"]
Jul 31 22:19:06.072856 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:19:06.072830    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92893c4d_43d6_42e4_a244_d7247e9acc44.slice/crio-c73b306be0f7935c0dfd5194b4d26e832a429eb5f1b81d398b787d6125bcf128 WatchSource:0}: Error finding container c73b306be0f7935c0dfd5194b4d26e832a429eb5f1b81d398b787d6125bcf128: Status 404 returned error can't find the container with id c73b306be0f7935c0dfd5194b4d26e832a429eb5f1b81d398b787d6125bcf128
Jul 31 22:19:06.075387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:06.075345    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:19:06.075473 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:06.075438    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:19:06.075510 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:06.075483    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:19:06.608268 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:06.608222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" event={"ID":"92893c4d-43d6-42e4-a244-d7247e9acc44","Type":"ContainerStarted","Data":"f3be53e34009d554c8a8e7d567b693ee239aad4db9b169f45dc10d7fcc23c3c7"}
Jul 31 22:19:06.608515 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:06.608296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" event={"ID":"92893c4d-43d6-42e4-a244-d7247e9acc44","Type":"ContainerStarted","Data":"c73b306be0f7935c0dfd5194b4d26e832a429eb5f1b81d398b787d6125bcf128"}
Jul 31 22:19:06.626957 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:06.626900    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" podStartSLOduration=1.6268741420000001 podStartE2EDuration="1.626874142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:19:06.625701837 +0000 UTC m=+2586.888450201" watchObservedRunningTime="2026-07-31 22:19:06.626874142 +0000 UTC m=+2586.889622509"
Jul 31 22:19:06.929309 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:06.929198    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:07.935475 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:07.935444    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:08.617463 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:08.617434    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:08.618439 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:08.618420    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:09.640832 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.640799    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"]
Jul 31 22:19:09.641277 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.641223    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy"
Jul 31 22:19:09.641277 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.641236    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy"
Jul 31 22:19:09.641368 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.641316    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="55db13a2-0a54-4fb5-b9ef-e671c2678da0" containerName="istio-proxy"
Jul 31 22:19:09.644373 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.644354    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:09.646326 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.646299    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ef8f59-serving-cert\""
Jul 31 22:19:09.654757 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.654728    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"]
Jul 31 22:19:09.783488 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.783435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5kv\" (UniqueName: \"kubernetes.io/projected/864967bf-1787-42b2-b4f4-0985613b036e-kube-api-access-lj5kv\") pod \"maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:09.783703 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.783523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864967bf-1787-42b2-b4f4-0985613b036e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:09.783776 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.783720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864967bf-1787-42b2-b4f4-0985613b036e-maas-api-tls\") pod \"maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:09.884282 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.884250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5kv\" (UniqueName: \"kubernetes.io/projected/864967bf-1787-42b2-b4f4-0985613b036e-kube-api-access-lj5kv\") pod \"maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:09.884282 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.884285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864967bf-1787-42b2-b4f4-0985613b036e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:09.884543 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.884323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864967bf-1787-42b2-b4f4-0985613b036e-maas-api-tls\") pod \"maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:09.885011 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.884983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864967bf-1787-42b2-b4f4-0985613b036e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:09.886859 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.886838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864967bf-1787-42b2-b4f4-0985613b036e-maas-api-tls\") pod \"maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:09.898421 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.898350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5kv\" (UniqueName: \"kubernetes.io/projected/864967bf-1787-42b2-b4f4-0985613b036e-kube-api-access-lj5kv\") pod \"maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:09.957483 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:09.957442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:10.133673 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.133644    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"]
Jul 31 22:19:10.136147 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:19:10.136099    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864967bf_1787_42b2_b4f4_0985613b036e.slice/crio-77a53d928efbca722b9520883e9d43f400d86db3a711f43cbd898cf3fdcaa1c5 WatchSource:0}: Error finding container 77a53d928efbca722b9520883e9d43f400d86db3a711f43cbd898cf3fdcaa1c5: Status 404 returned error can't find the container with id 77a53d928efbca722b9520883e9d43f400d86db3a711f43cbd898cf3fdcaa1c5
Jul 31 22:19:10.137799 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.137774    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 22:19:10.389680 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.389640    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"]
Jul 31 22:19:10.394454 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.394432    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:19:10.396340 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.396314    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ef8f59\""
Jul 31 22:19:10.396478 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.396409    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ef8f59-dockercfg-lnjsk\""
Jul 31 22:19:10.400805 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.400775    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"]
Jul 31 22:19:10.440631 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.440590    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"]
Jul 31 22:19:10.444188 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.444168    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:19:10.455063 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.455031    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"]
Jul 31 22:19:10.489503 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.489462    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zk2\" (UniqueName: \"kubernetes.io/projected/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-kube-api-access-l4zk2\") pod \"payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6\" (UID: \"b66ad9f1-05a0-4ae4-aea5-45d6832af67b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:19:10.489681 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.489544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6\" (UID: \"b66ad9f1-05a0-4ae4-aea5-45d6832af67b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:19:10.590372 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.590329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh4pf\" (UniqueName: \"kubernetes.io/projected/d21b84d5-ad18-407f-8775-05e01f04762e-kube-api-access-bh4pf\") pod \"payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv\" (UID: \"d21b84d5-ad18-407f-8775-05e01f04762e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:19:10.590538 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.590482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21b84d5-ad18-407f-8775-05e01f04762e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv\" (UID: \"d21b84d5-ad18-407f-8775-05e01f04762e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:19:10.590538 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.590516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zk2\" (UniqueName: \"kubernetes.io/projected/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-kube-api-access-l4zk2\") pod \"payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6\" (UID: \"b66ad9f1-05a0-4ae4-aea5-45d6832af67b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:19:10.590635 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.590545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6\" (UID: \"b66ad9f1-05a0-4ae4-aea5-45d6832af67b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:19:10.591227 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.591210    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6\" (UID: \"b66ad9f1-05a0-4ae4-aea5-45d6832af67b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:19:10.598104 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.598070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zk2\" (UniqueName: \"kubernetes.io/projected/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-kube-api-access-l4zk2\") pod \"payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6\" (UID: \"b66ad9f1-05a0-4ae4-aea5-45d6832af67b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:19:10.629499 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.629459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g" event={"ID":"864967bf-1787-42b2-b4f4-0985613b036e","Type":"ContainerStarted","Data":"77a53d928efbca722b9520883e9d43f400d86db3a711f43cbd898cf3fdcaa1c5"}
Jul 31 22:19:10.691357 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.691265    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh4pf\" (UniqueName: \"kubernetes.io/projected/d21b84d5-ad18-407f-8775-05e01f04762e-kube-api-access-bh4pf\") pod \"payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv\" (UID: \"d21b84d5-ad18-407f-8775-05e01f04762e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:19:10.691831 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.691433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21b84d5-ad18-407f-8775-05e01f04762e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv\" (UID: \"d21b84d5-ad18-407f-8775-05e01f04762e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:19:10.692080 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.692060    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21b84d5-ad18-407f-8775-05e01f04762e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv\" (UID: \"d21b84d5-ad18-407f-8775-05e01f04762e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:19:10.698659 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.698630    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh4pf\" (UniqueName: \"kubernetes.io/projected/d21b84d5-ad18-407f-8775-05e01f04762e-kube-api-access-bh4pf\") pod \"payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv\" (UID: \"d21b84d5-ad18-407f-8775-05e01f04762e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:19:10.708528 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.708496    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:19:10.755186 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.755157    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:19:10.867872 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.867840    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"]
Jul 31 22:19:10.870497 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:19:10.870467    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66ad9f1_05a0_4ae4_aea5_45d6832af67b.slice/crio-a0fc3f77b1c8c53232c1e58099fdb082ea226e79a0356c3ae38141ae49095569 WatchSource:0}: Error finding container a0fc3f77b1c8c53232c1e58099fdb082ea226e79a0356c3ae38141ae49095569: Status 404 returned error can't find the container with id a0fc3f77b1c8c53232c1e58099fdb082ea226e79a0356c3ae38141ae49095569
Jul 31 22:19:10.915624 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:10.915595    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"]
Jul 31 22:19:10.925971 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:19:10.925857    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21b84d5_ad18_407f_8775_05e01f04762e.slice/crio-186e9359d455491408612ee7578ed337faaae0944cddd2992c260d80cd27cfad WatchSource:0}: Error finding container 186e9359d455491408612ee7578ed337faaae0944cddd2992c260d80cd27cfad: Status 404 returned error can't find the container with id 186e9359d455491408612ee7578ed337faaae0944cddd2992c260d80cd27cfad
Jul 31 22:19:11.636398 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:11.636353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6" event={"ID":"b66ad9f1-05a0-4ae4-aea5-45d6832af67b","Type":"ContainerStarted","Data":"1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344"}
Jul 31 22:19:11.636709 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:11.636420    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6" event={"ID":"b66ad9f1-05a0-4ae4-aea5-45d6832af67b","Type":"ContainerStarted","Data":"a0fc3f77b1c8c53232c1e58099fdb082ea226e79a0356c3ae38141ae49095569"}
Jul 31 22:19:11.636709 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:11.636534    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:19:11.638158 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:11.638106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g" event={"ID":"864967bf-1787-42b2-b4f4-0985613b036e","Type":"ContainerStarted","Data":"99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62"}
Jul 31 22:19:11.638303 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:11.638228    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:11.639803 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:11.639776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv" event={"ID":"d21b84d5-ad18-407f-8775-05e01f04762e","Type":"ContainerStarted","Data":"e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f"}
Jul 31 22:19:11.639939 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:11.639824    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv" event={"ID":"d21b84d5-ad18-407f-8775-05e01f04762e","Type":"ContainerStarted","Data":"186e9359d455491408612ee7578ed337faaae0944cddd2992c260d80cd27cfad"}
Jul 31 22:19:11.639939 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:11.639847    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:19:11.651039 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:11.650985    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6" podStartSLOduration=1.650967589 podStartE2EDuration="1.650967589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:19:11.649863232 +0000 UTC m=+2591.912611597" watchObservedRunningTime="2026-07-31 22:19:11.650967589 +0000 UTC m=+2591.913715952"
Jul 31 22:19:11.667085 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:11.667027    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g" podStartSLOduration=2.088930054 podStartE2EDuration="2.667006693s" totalImagesPullingTime="578.076639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:19:11.664793828 +0000 UTC m=+2591.927542194" watchObservedRunningTime="2026-07-31 22:19:11.667006693 +0000 UTC m=+2591.929755056"
Jul 31 22:19:11.679161 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:11.679048    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv" podStartSLOduration=1.679018726 podStartE2EDuration="1.679018726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:19:11.678563696 +0000 UTC m=+2591.941312061" watchObservedRunningTime="2026-07-31 22:19:11.679018726 +0000 UTC m=+2591.941767091"
Jul 31 22:19:17.651335 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:17.651301    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:19:22.646624 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:22.646579    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv" podUID="d21b84d5-ad18-407f-8775-05e01f04762e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.68:9004: connect: connection refused"
Jul 31 22:19:22.647171 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:22.647103    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:19:22.707952 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:22.707864    2561 generic.go:356] "Generic (PLEG): container finished" podID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerID="f3be53e34009d554c8a8e7d567b693ee239aad4db9b169f45dc10d7fcc23c3c7" exitCode=0
Jul 31 22:19:22.707952 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:22.707933    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" event={"ID":"92893c4d-43d6-42e4-a244-d7247e9acc44","Type":"ContainerDied","Data":"f3be53e34009d554c8a8e7d567b693ee239aad4db9b169f45dc10d7fcc23c3c7"}
Jul 31 22:19:22.708390 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:22.708374    2561 scope.go:117] "RemoveContainer" containerID="f3be53e34009d554c8a8e7d567b693ee239aad4db9b169f45dc10d7fcc23c3c7"
Jul 31 22:19:22.709533 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:22.709494    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:19:22.709608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:22.709592    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:19:22.709651 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:22.709640    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:19:23.617658 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:23.617618    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:23.718190 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:23.718150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" event={"ID":"92893c4d-43d6-42e4-a244-d7247e9acc44","Type":"ContainerStarted","Data":"459fcc5637ded75aebdf001ad6ddb7eed9c63e808e7706c2cea2ce18ec700737"}
Jul 31 22:19:23.929344 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:23.929255    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:23.930282 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:23.930258    2561 patch_prober.go:28] interesting pod/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 31 22:19:23.930344 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:23.930310    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 31 22:19:24.930419 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:24.930374    2561 patch_prober.go:28] interesting pod/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 31 22:19:24.930810 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:24.930449    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 31 22:19:25.930515 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:25.930472    2561 patch_prober.go:28] interesting pod/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 31 22:19:25.930912 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:25.930534    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 31 22:19:26.929555 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:26.929518    2561 patch_prober.go:28] interesting pod/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 31 22:19:26.929746 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:26.929573    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 31 22:19:27.930553 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:27.930512    2561 patch_prober.go:28] interesting pod/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 31 22:19:27.931065 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:27.930583    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 31 22:19:28.934091 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:28.934054    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:28.934505 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:28.934465    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:28.935097 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:28.935081    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:19:32.646339 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:32.646296    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:19:38.733670 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.733630    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"]
Jul 31 22:19:38.740439 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.740406    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:38.744197 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.744167    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d193d8-serving-cert\""
Jul 31 22:19:38.749569 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.749538    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"]
Jul 31 22:19:38.759519 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.759488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68mdb\" (UniqueName: \"kubernetes.io/projected/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-kube-api-access-68mdb\") pod \"maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:38.759697 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.759533    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-maas-api-tls\") pod \"maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:38.759834 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.759705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-openshift-service-ca\") pod \"maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:38.860718 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.860674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-maas-api-tls\") pod \"maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:38.860932 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.860807    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-openshift-service-ca\") pod \"maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:38.860932 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:19:38.860841    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-serving-cert: secret "maas-api-e2e-api-b-d193d8-serving-cert" not found
Jul 31 22:19:38.860932 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.860867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68mdb\" (UniqueName: \"kubernetes.io/projected/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-kube-api-access-68mdb\") pod \"maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:38.860932 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:19:38.860912    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-maas-api-tls podName:7ddaf908-ad53-4b1b-aa10-c3d3f582860c nodeName:}" failed. No retries permitted until 2026-07-31 22:19:39.360892657 +0000 UTC m=+2619.623641004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-maas-api-tls") pod "maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb" (UID: "7ddaf908-ad53-4b1b-aa10-c3d3f582860c") : secret "maas-api-e2e-api-b-d193d8-serving-cert" not found
Jul 31 22:19:38.861540 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.861518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-openshift-service-ca\") pod \"maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:38.871344 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:38.871309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68mdb\" (UniqueName: \"kubernetes.io/projected/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-kube-api-access-68mdb\") pod \"maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:39.365089 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.365050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-maas-api-tls\") pod \"maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:39.367496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.367469    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-maas-api-tls\") pod \"maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:39.396161 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.396095    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"]
Jul 31 22:19:39.399961 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.399935    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:19:39.402093 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.402063    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d193d8\""
Jul 31 22:19:39.402254 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.402163    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d193d8-dockercfg-r4llf\""
Jul 31 22:19:39.408571 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.408547    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"]
Jul 31 22:19:39.425513 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.425484    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"]
Jul 31 22:19:39.429196 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.429163    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:19:39.439713 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.439677    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"]
Jul 31 22:19:39.466012 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.465978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz\" (UID: \"8d87eb1f-02bb-45b2-9c54-6144aebe2e28\") " pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:19:39.466217 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.466027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dpgc\" (UniqueName: \"kubernetes.io/projected/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-kube-api-access-4dpgc\") pod \"payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz\" (UID: \"8d87eb1f-02bb-45b2-9c54-6144aebe2e28\") " pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:19:39.466217 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.466054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5v7\" (UniqueName: \"kubernetes.io/projected/3ca12171-aeba-486b-91b9-e9141bca239b-kube-api-access-xs5v7\") pod \"payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4\" (UID: \"3ca12171-aeba-486b-91b9-e9141bca239b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:19:39.466217 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.466163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca12171-aeba-486b-91b9-e9141bca239b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4\" (UID: \"3ca12171-aeba-486b-91b9-e9141bca239b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:19:39.567258 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.567213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5v7\" (UniqueName: \"kubernetes.io/projected/3ca12171-aeba-486b-91b9-e9141bca239b-kube-api-access-xs5v7\") pod \"payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4\" (UID: \"3ca12171-aeba-486b-91b9-e9141bca239b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:19:39.567475 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.567284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca12171-aeba-486b-91b9-e9141bca239b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4\" (UID: \"3ca12171-aeba-486b-91b9-e9141bca239b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:19:39.567475 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.567391    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz\" (UID: \"8d87eb1f-02bb-45b2-9c54-6144aebe2e28\") " pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:19:39.567475 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.567437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dpgc\" (UniqueName: \"kubernetes.io/projected/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-kube-api-access-4dpgc\") pod \"payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz\" (UID: \"8d87eb1f-02bb-45b2-9c54-6144aebe2e28\") " pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:19:39.568273 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.568245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca12171-aeba-486b-91b9-e9141bca239b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4\" (UID: \"3ca12171-aeba-486b-91b9-e9141bca239b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:19:39.568370 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.568248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz\" (UID: \"8d87eb1f-02bb-45b2-9c54-6144aebe2e28\") " pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:19:39.575076 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.575043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5v7\" (UniqueName: \"kubernetes.io/projected/3ca12171-aeba-486b-91b9-e9141bca239b-kube-api-access-xs5v7\") pod \"payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4\" (UID: \"3ca12171-aeba-486b-91b9-e9141bca239b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:19:39.575252 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.575229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dpgc\" (UniqueName: \"kubernetes.io/projected/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-kube-api-access-4dpgc\") pod \"payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz\" (UID: \"8d87eb1f-02bb-45b2-9c54-6144aebe2e28\") " pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:19:39.658729 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.658634    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:39.712733 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.712300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:19:39.742469 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.742224    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:19:39.822890 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.822838    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"]
Jul 31 22:19:39.827661 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:19:39.826422    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddaf908_ad53_4b1b_aa10_c3d3f582860c.slice/crio-fd99af72ae9011aa280a660c8fc8810f5aa348e346bbf17d2cde6ec8e9def46d WatchSource:0}: Error finding container fd99af72ae9011aa280a660c8fc8810f5aa348e346bbf17d2cde6ec8e9def46d: Status 404 returned error can't find the container with id fd99af72ae9011aa280a660c8fc8810f5aa348e346bbf17d2cde6ec8e9def46d
Jul 31 22:19:39.889083 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.888951    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"]
Jul 31 22:19:39.921716 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:39.921687    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"]
Jul 31 22:19:39.925121 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:19:39.925093    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca12171_aeba_486b_91b9_e9141bca239b.slice/crio-90af78143dc8b1072ebed09c1009cf361ca286df62ea92623312866b3395b154 WatchSource:0}: Error finding container 90af78143dc8b1072ebed09c1009cf361ca286df62ea92623312866b3395b154: Status 404 returned error can't find the container with id 90af78143dc8b1072ebed09c1009cf361ca286df62ea92623312866b3395b154
Jul 31 22:19:40.815448 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.815414    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4" event={"ID":"3ca12171-aeba-486b-91b9-e9141bca239b","Type":"ContainerStarted","Data":"157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d"}
Jul 31 22:19:40.815989 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.815467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4" event={"ID":"3ca12171-aeba-486b-91b9-e9141bca239b","Type":"ContainerStarted","Data":"90af78143dc8b1072ebed09c1009cf361ca286df62ea92623312866b3395b154"}
Jul 31 22:19:40.815989 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.815517    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:19:40.817201 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.817164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb" event={"ID":"7ddaf908-ad53-4b1b-aa10-c3d3f582860c","Type":"ContainerStarted","Data":"daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0"}
Jul 31 22:19:40.817350 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.817227    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb" event={"ID":"7ddaf908-ad53-4b1b-aa10-c3d3f582860c","Type":"ContainerStarted","Data":"fd99af72ae9011aa280a660c8fc8810f5aa348e346bbf17d2cde6ec8e9def46d"}
Jul 31 22:19:40.817350 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.817259    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:40.818663 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.818638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz" event={"ID":"8d87eb1f-02bb-45b2-9c54-6144aebe2e28","Type":"ContainerStarted","Data":"bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03"}
Jul 31 22:19:40.818772 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.818686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz" event={"ID":"8d87eb1f-02bb-45b2-9c54-6144aebe2e28","Type":"ContainerStarted","Data":"2fc21b35c5729c59e720a0e8fd93dfe75686ece00956d053c985d5a23b2f0dad"}
Jul 31 22:19:40.818772 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.818728    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:19:40.829489 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.829438    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4" podStartSLOduration=1.829418028 podStartE2EDuration="1.829418028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:19:40.82900051 +0000 UTC m=+2621.091748868" watchObservedRunningTime="2026-07-31 22:19:40.829418028 +0000 UTC m=+2621.092166392"
Jul 31 22:19:40.845497 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.845442    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb" podStartSLOduration=2.423215513 podStartE2EDuration="2.845419863s" totalImagesPullingTime="422.20435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:19:40.842892695 +0000 UTC m=+2621.105641060" watchObservedRunningTime="2026-07-31 22:19:40.845419863 +0000 UTC m=+2621.108168227"
Jul 31 22:19:40.857963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:40.857889    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz" podStartSLOduration=1.857868498 podStartE2EDuration="1.857868498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:19:40.85711045 +0000 UTC m=+2621.119858813" watchObservedRunningTime="2026-07-31 22:19:40.857868498 +0000 UTC m=+2621.120616862"
Jul 31 22:19:46.830243 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:46.830209    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:19:51.826731 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:51.826697    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:19:51.827224 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:19:51.826789    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:20:21.117350 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.117308    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"]
Jul 31 22:20:21.117995 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.117587    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g" podUID="864967bf-1787-42b2-b4f4-0985613b036e" containerName="maas-api" containerID="cri-o://99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62" gracePeriod=30
Jul 31 22:20:21.175023 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.174985    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"]
Jul 31 22:20:21.175494 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.175297    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6" podUID="b66ad9f1-05a0-4ae4-aea5-45d6832af67b" containerName="payload-processing" containerID="cri-o://1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344" gracePeriod=30
Jul 31 22:20:21.187241 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.187198    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"]
Jul 31 22:20:21.187449 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.187409    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv" podUID="d21b84d5-ad18-407f-8775-05e01f04762e" containerName="payload-pre-processing" containerID="cri-o://e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f" gracePeriod=30
Jul 31 22:20:21.464953 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.464916    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:20:21.498817 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.498791    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:20:21.505411 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.505379    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:20:21.564180 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.564114    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh4pf\" (UniqueName: \"kubernetes.io/projected/d21b84d5-ad18-407f-8775-05e01f04762e-kube-api-access-bh4pf\") pod \"d21b84d5-ad18-407f-8775-05e01f04762e\" (UID: \"d21b84d5-ad18-407f-8775-05e01f04762e\") "
Jul 31 22:20:21.564387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.564229    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4zk2\" (UniqueName: \"kubernetes.io/projected/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-kube-api-access-l4zk2\") pod \"b66ad9f1-05a0-4ae4-aea5-45d6832af67b\" (UID: \"b66ad9f1-05a0-4ae4-aea5-45d6832af67b\") "
Jul 31 22:20:21.564387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.564255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj5kv\" (UniqueName: \"kubernetes.io/projected/864967bf-1787-42b2-b4f4-0985613b036e-kube-api-access-lj5kv\") pod \"864967bf-1787-42b2-b4f4-0985613b036e\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") "
Jul 31 22:20:21.564387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.564277    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21b84d5-ad18-407f-8775-05e01f04762e-plugins-config-volume\") pod \"d21b84d5-ad18-407f-8775-05e01f04762e\" (UID: \"d21b84d5-ad18-407f-8775-05e01f04762e\") "
Jul 31 22:20:21.564387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.564305    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864967bf-1787-42b2-b4f4-0985613b036e-maas-api-tls\") pod \"864967bf-1787-42b2-b4f4-0985613b036e\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") "
Jul 31 22:20:21.564387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.564335    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864967bf-1787-42b2-b4f4-0985613b036e-openshift-service-ca\") pod \"864967bf-1787-42b2-b4f4-0985613b036e\" (UID: \"864967bf-1787-42b2-b4f4-0985613b036e\") "
Jul 31 22:20:21.564387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.564354    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-plugins-config-volume\") pod \"b66ad9f1-05a0-4ae4-aea5-45d6832af67b\" (UID: \"b66ad9f1-05a0-4ae4-aea5-45d6832af67b\") "
Jul 31 22:20:21.564759 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.564723    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864967bf-1787-42b2-b4f4-0985613b036e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "864967bf-1787-42b2-b4f4-0985613b036e" (UID: "864967bf-1787-42b2-b4f4-0985613b036e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:20:21.564818 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.564733    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21b84d5-ad18-407f-8775-05e01f04762e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d21b84d5-ad18-407f-8775-05e01f04762e" (UID: "d21b84d5-ad18-407f-8775-05e01f04762e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:20:21.564875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.564820    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b66ad9f1-05a0-4ae4-aea5-45d6832af67b" (UID: "b66ad9f1-05a0-4ae4-aea5-45d6832af67b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:20:21.566547 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.566518    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864967bf-1787-42b2-b4f4-0985613b036e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "864967bf-1787-42b2-b4f4-0985613b036e" (UID: "864967bf-1787-42b2-b4f4-0985613b036e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:20:21.566670 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.566568    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21b84d5-ad18-407f-8775-05e01f04762e-kube-api-access-bh4pf" (OuterVolumeSpecName: "kube-api-access-bh4pf") pod "d21b84d5-ad18-407f-8775-05e01f04762e" (UID: "d21b84d5-ad18-407f-8775-05e01f04762e"). InnerVolumeSpecName "kube-api-access-bh4pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:20:21.566670 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.566603    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-kube-api-access-l4zk2" (OuterVolumeSpecName: "kube-api-access-l4zk2") pod "b66ad9f1-05a0-4ae4-aea5-45d6832af67b" (UID: "b66ad9f1-05a0-4ae4-aea5-45d6832af67b"). InnerVolumeSpecName "kube-api-access-l4zk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:20:21.566670 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.566633    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864967bf-1787-42b2-b4f4-0985613b036e-kube-api-access-lj5kv" (OuterVolumeSpecName: "kube-api-access-lj5kv") pod "864967bf-1787-42b2-b4f4-0985613b036e" (UID: "864967bf-1787-42b2-b4f4-0985613b036e"). InnerVolumeSpecName "kube-api-access-lj5kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:20:21.665353 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.665316    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4zk2\" (UniqueName: \"kubernetes.io/projected/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-kube-api-access-l4zk2\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:21.665353 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.665349    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj5kv\" (UniqueName: \"kubernetes.io/projected/864967bf-1787-42b2-b4f4-0985613b036e-kube-api-access-lj5kv\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:21.665353 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.665361    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21b84d5-ad18-407f-8775-05e01f04762e-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:21.665619 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.665374    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864967bf-1787-42b2-b4f4-0985613b036e-maas-api-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:21.665619 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.665388    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864967bf-1787-42b2-b4f4-0985613b036e-openshift-service-ca\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:21.665619 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.665400    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66ad9f1-05a0-4ae4-aea5-45d6832af67b-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:21.665619 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:21.665414    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh4pf\" (UniqueName: \"kubernetes.io/projected/d21b84d5-ad18-407f-8775-05e01f04762e-kube-api-access-bh4pf\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:22.030575 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.030540    2561 generic.go:356] "Generic (PLEG): container finished" podID="864967bf-1787-42b2-b4f4-0985613b036e" containerID="99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62" exitCode=0
Jul 31 22:20:22.030808 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.030587    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"
Jul 31 22:20:22.030808 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.030615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g" event={"ID":"864967bf-1787-42b2-b4f4-0985613b036e","Type":"ContainerDied","Data":"99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62"}
Jul 31 22:20:22.030808 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.030667    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g" event={"ID":"864967bf-1787-42b2-b4f4-0985613b036e","Type":"ContainerDied","Data":"77a53d928efbca722b9520883e9d43f400d86db3a711f43cbd898cf3fdcaa1c5"}
Jul 31 22:20:22.030808 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.030699    2561 scope.go:117] "RemoveContainer" containerID="99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62"
Jul 31 22:20:22.032237 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.032201    2561 generic.go:356] "Generic (PLEG): container finished" podID="d21b84d5-ad18-407f-8775-05e01f04762e" containerID="e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f" exitCode=0
Jul 31 22:20:22.032369 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.032258    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"
Jul 31 22:20:22.032369 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.032321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv" event={"ID":"d21b84d5-ad18-407f-8775-05e01f04762e","Type":"ContainerDied","Data":"e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f"}
Jul 31 22:20:22.032486 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.032366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv" event={"ID":"d21b84d5-ad18-407f-8775-05e01f04762e","Type":"ContainerDied","Data":"186e9359d455491408612ee7578ed337faaae0944cddd2992c260d80cd27cfad"}
Jul 31 22:20:22.033812 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.033780    2561 generic.go:356] "Generic (PLEG): container finished" podID="b66ad9f1-05a0-4ae4-aea5-45d6832af67b" containerID="1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344" exitCode=0
Jul 31 22:20:22.033918 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.033808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6" event={"ID":"b66ad9f1-05a0-4ae4-aea5-45d6832af67b","Type":"ContainerDied","Data":"1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344"}
Jul 31 22:20:22.033918 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.033842    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"
Jul 31 22:20:22.033918 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.033853    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6" event={"ID":"b66ad9f1-05a0-4ae4-aea5-45d6832af67b","Type":"ContainerDied","Data":"a0fc3f77b1c8c53232c1e58099fdb082ea226e79a0356c3ae38141ae49095569"}
Jul 31 22:20:22.043670 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.043020    2561 scope.go:117] "RemoveContainer" containerID="99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62"
Jul 31 22:20:22.048478 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:20:22.044480    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62\": container with ID starting with 99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62 not found: ID does not exist" containerID="99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62"
Jul 31 22:20:22.048478 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.044531    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62"} err="failed to get container status \"99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62\": rpc error: code = NotFound desc = could not find container \"99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62\": container with ID starting with 99da94dabf4a8f0e530e83f268f1241d154b50917699cda5f780f1b7cae2ed62 not found: ID does not exist"
Jul 31 22:20:22.048478 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.044559    2561 scope.go:117] "RemoveContainer" containerID="e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f"
Jul 31 22:20:22.058607 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.058581    2561 scope.go:117] "RemoveContainer" containerID="e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f"
Jul 31 22:20:22.059262 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:20:22.058985    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f\": container with ID starting with e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f not found: ID does not exist" containerID="e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f"
Jul 31 22:20:22.059262 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.059029    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f"} err="failed to get container status \"e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f\": rpc error: code = NotFound desc = could not find container \"e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f\": container with ID starting with e58bc5510fab5e511b125823a677e68c99ddaf254568497c98566a239a10b20f not found: ID does not exist"
Jul 31 22:20:22.059262 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.059057    2561 scope.go:117] "RemoveContainer" containerID="1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344"
Jul 31 22:20:22.061353 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.061330    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"]
Jul 31 22:20:22.064357 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.064335    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef8f59-6f9d9589d9-77q4g"]
Jul 31 22:20:22.067989 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.067965    2561 scope.go:117] "RemoveContainer" containerID="1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344"
Jul 31 22:20:22.068356 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:20:22.068337    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344\": container with ID starting with 1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344 not found: ID does not exist" containerID="1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344"
Jul 31 22:20:22.068436 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.068366    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344"} err="failed to get container status \"1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344\": rpc error: code = NotFound desc = could not find container \"1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344\": container with ID starting with 1399c92141d3b72b917827e609e19add812ddbfa1d4ff4bee4070300e84ef344 not found: ID does not exist"
Jul 31 22:20:22.073044 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.073017    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"]
Jul 31 22:20:22.075098 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.075076    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef8f59-6494b7d9f6-g4xfv"]
Jul 31 22:20:22.086408 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.086377    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"]
Jul 31 22:20:22.088301 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.088279    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef8f59-5b7bdb8f4-sg6z6"]
Jul 31 22:20:22.227913 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.227877    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="864967bf-1787-42b2-b4f4-0985613b036e" path="/var/lib/kubelet/pods/864967bf-1787-42b2-b4f4-0985613b036e/volumes"
Jul 31 22:20:22.228326 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.228291    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b66ad9f1-05a0-4ae4-aea5-45d6832af67b" path="/var/lib/kubelet/pods/b66ad9f1-05a0-4ae4-aea5-45d6832af67b/volumes"
Jul 31 22:20:22.228623 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:22.228610    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d21b84d5-ad18-407f-8775-05e01f04762e" path="/var/lib/kubelet/pods/d21b84d5-ad18-407f-8775-05e01f04762e/volumes"
Jul 31 22:20:27.833045 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:27.832994    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"]
Jul 31 22:20:27.833885 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:27.833268    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy" containerID="cri-o://459fcc5637ded75aebdf001ad6ddb7eed9c63e808e7706c2cea2ce18ec700737" gracePeriod=30
Jul 31 22:20:28.937123 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:28.937086    2561 patch_prober.go:28] interesting pod/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 31 22:20:28.937543 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:28.937181    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 22:20:33.096216 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.096185    2561 generic.go:356] "Generic (PLEG): container finished" podID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerID="459fcc5637ded75aebdf001ad6ddb7eed9c63e808e7706c2cea2ce18ec700737" exitCode=0
Jul 31 22:20:33.096568 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.096261    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" event={"ID":"92893c4d-43d6-42e4-a244-d7247e9acc44","Type":"ContainerDied","Data":"459fcc5637ded75aebdf001ad6ddb7eed9c63e808e7706c2cea2ce18ec700737"}
Jul 31 22:20:33.096568 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.096319    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk" event={"ID":"92893c4d-43d6-42e4-a244-d7247e9acc44","Type":"ContainerDied","Data":"c73b306be0f7935c0dfd5194b4d26e832a429eb5f1b81d398b787d6125bcf128"}
Jul 31 22:20:33.096568 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.096338    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c73b306be0f7935c0dfd5194b4d26e832a429eb5f1b81d398b787d6125bcf128"
Jul 31 22:20:33.096568 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.096353    2561 scope.go:117] "RemoveContainer" containerID="f3be53e34009d554c8a8e7d567b693ee239aad4db9b169f45dc10d7fcc23c3c7"
Jul 31 22:20:33.101711 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.101645    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:20:33.176935 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.176893    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-podinfo\") pod \"92893c4d-43d6-42e4-a244-d7247e9acc44\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") "
Jul 31 22:20:33.176935 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.176945    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-socket\") pod \"92893c4d-43d6-42e4-a244-d7247e9acc44\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") "
Jul 31 22:20:33.177261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.176967    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-data\") pod \"92893c4d-43d6-42e4-a244-d7247e9acc44\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") "
Jul 31 22:20:33.177261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.176995    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcjpb\" (UniqueName: \"kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-kube-api-access-lcjpb\") pod \"92893c4d-43d6-42e4-a244-d7247e9acc44\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") "
Jul 31 22:20:33.177261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177012    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-token\") pod \"92893c4d-43d6-42e4-a244-d7247e9acc44\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") "
Jul 31 22:20:33.177261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177028    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92893c4d-43d6-42e4-a244-d7247e9acc44-istiod-ca-cert\") pod \"92893c4d-43d6-42e4-a244-d7247e9acc44\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") "
Jul 31 22:20:33.177261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-credential-socket\") pod \"92893c4d-43d6-42e4-a244-d7247e9acc44\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") "
Jul 31 22:20:33.177261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177193    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-certs\") pod \"92893c4d-43d6-42e4-a244-d7247e9acc44\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") "
Jul 31 22:20:33.177261 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177240    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92893c4d-43d6-42e4-a244-d7247e9acc44" (UID: "92893c4d-43d6-42e4-a244-d7247e9acc44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:20:33.177611 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177398    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-data" (OuterVolumeSpecName: "istio-data") pod "92893c4d-43d6-42e4-a244-d7247e9acc44" (UID: "92893c4d-43d6-42e4-a244-d7247e9acc44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:20:33.177611 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177441    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92893c4d-43d6-42e4-a244-d7247e9acc44" (UID: "92893c4d-43d6-42e4-a244-d7247e9acc44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:20:33.177611 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177532    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92893c4d-43d6-42e4-a244-d7247e9acc44" (UID: "92893c4d-43d6-42e4-a244-d7247e9acc44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:20:33.177611 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177556    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92893c4d-43d6-42e4-a244-d7247e9acc44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92893c4d-43d6-42e4-a244-d7247e9acc44" (UID: "92893c4d-43d6-42e4-a244-d7247e9acc44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:20:33.177824 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177682    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-envoy\") pod \"92893c4d-43d6-42e4-a244-d7247e9acc44\" (UID: \"92893c4d-43d6-42e4-a244-d7247e9acc44\") "
Jul 31 22:20:33.177952 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177936    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-data\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:33.177952 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177952    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92893c4d-43d6-42e4-a244-d7247e9acc44-istiod-ca-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:33.178073 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177961    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-credential-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:33.178073 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177971    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:33.178073 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.177982    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-workload-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:33.179414 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.179389    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92893c4d-43d6-42e4-a244-d7247e9acc44" (UID: "92893c4d-43d6-42e4-a244-d7247e9acc44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 22:20:33.179527 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.179432    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-token" (OuterVolumeSpecName: "istio-token") pod "92893c4d-43d6-42e4-a244-d7247e9acc44" (UID: "92893c4d-43d6-42e4-a244-d7247e9acc44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:20:33.179527 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.179450    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-kube-api-access-lcjpb" (OuterVolumeSpecName: "kube-api-access-lcjpb") pod "92893c4d-43d6-42e4-a244-d7247e9acc44" (UID: "92893c4d-43d6-42e4-a244-d7247e9acc44"). InnerVolumeSpecName "kube-api-access-lcjpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:20:33.179527 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.179520    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92893c4d-43d6-42e4-a244-d7247e9acc44" (UID: "92893c4d-43d6-42e4-a244-d7247e9acc44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:20:33.279454 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.279410    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-podinfo\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:33.279454 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.279448    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcjpb\" (UniqueName: \"kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-kube-api-access-lcjpb\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:33.279454 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.279459    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:33.279454 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:33.279467    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92893c4d-43d6-42e4-a244-d7247e9acc44-istio-envoy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:34.102702 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:34.102667    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"
Jul 31 22:20:34.133077 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:34.133038    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"]
Jul 31 22:20:34.135868 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:34.135843    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ef8f59-openshift-default-595f5f98c8-jwjqk"]
Jul 31 22:20:34.229196 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:34.229163    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" path="/var/lib/kubelet/pods/92893c4d-43d6-42e4-a244-d7247e9acc44/volumes"
Jul 31 22:20:43.585083 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.585039    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"]
Jul 31 22:20:43.585590 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.585344    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb" podUID="7ddaf908-ad53-4b1b-aa10-c3d3f582860c" containerName="maas-api" containerID="cri-o://daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0" gracePeriod=30
Jul 31 22:20:43.648723 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.648682    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"]
Jul 31 22:20:43.648942 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.648913    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz" podUID="8d87eb1f-02bb-45b2-9c54-6144aebe2e28" containerName="payload-processing" containerID="cri-o://bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03" gracePeriod=30
Jul 31 22:20:43.652276 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.652246    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"]
Jul 31 22:20:43.652530 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.652473    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4" podUID="3ca12171-aeba-486b-91b9-e9141bca239b" containerName="payload-pre-processing" containerID="cri-o://157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d" gracePeriod=30
Jul 31 22:20:43.908914 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.908880    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:20:43.946526 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.946495    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:20:43.963748 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.963717    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:20:43.980522 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.980488    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-maas-api-tls\") pod \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") "
Jul 31 22:20:43.981742 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.980540    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68mdb\" (UniqueName: \"kubernetes.io/projected/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-kube-api-access-68mdb\") pod \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") "
Jul 31 22:20:43.981742 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.980632    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-openshift-service-ca\") pod \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\" (UID: \"7ddaf908-ad53-4b1b-aa10-c3d3f582860c\") "
Jul 31 22:20:43.981742 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.981268    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ddaf908-ad53-4b1b-aa10-c3d3f582860c" (UID: "7ddaf908-ad53-4b1b-aa10-c3d3f582860c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:20:43.984046 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.984003    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ddaf908-ad53-4b1b-aa10-c3d3f582860c" (UID: "7ddaf908-ad53-4b1b-aa10-c3d3f582860c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:20:43.984220 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:43.984168    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-kube-api-access-68mdb" (OuterVolumeSpecName: "kube-api-access-68mdb") pod "7ddaf908-ad53-4b1b-aa10-c3d3f582860c" (UID: "7ddaf908-ad53-4b1b-aa10-c3d3f582860c"). InnerVolumeSpecName "kube-api-access-68mdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:20:44.081678 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.081639    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs5v7\" (UniqueName: \"kubernetes.io/projected/3ca12171-aeba-486b-91b9-e9141bca239b-kube-api-access-xs5v7\") pod \"3ca12171-aeba-486b-91b9-e9141bca239b\" (UID: \"3ca12171-aeba-486b-91b9-e9141bca239b\") "
Jul 31 22:20:44.081881 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.081750    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-plugins-config-volume\") pod \"8d87eb1f-02bb-45b2-9c54-6144aebe2e28\" (UID: \"8d87eb1f-02bb-45b2-9c54-6144aebe2e28\") "
Jul 31 22:20:44.081881 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.081773    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dpgc\" (UniqueName: \"kubernetes.io/projected/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-kube-api-access-4dpgc\") pod \"8d87eb1f-02bb-45b2-9c54-6144aebe2e28\" (UID: \"8d87eb1f-02bb-45b2-9c54-6144aebe2e28\") "
Jul 31 22:20:44.081881 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.081825    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca12171-aeba-486b-91b9-e9141bca239b-plugins-config-volume\") pod \"3ca12171-aeba-486b-91b9-e9141bca239b\" (UID: \"3ca12171-aeba-486b-91b9-e9141bca239b\") "
Jul 31 22:20:44.082113 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.082095    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-maas-api-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:44.082224 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.082117    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68mdb\" (UniqueName: \"kubernetes.io/projected/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-kube-api-access-68mdb\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:44.082224 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.082170    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddaf908-ad53-4b1b-aa10-c3d3f582860c-openshift-service-ca\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:44.082224 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.082184    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d87eb1f-02bb-45b2-9c54-6144aebe2e28" (UID: "8d87eb1f-02bb-45b2-9c54-6144aebe2e28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:20:44.082327 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.082239    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca12171-aeba-486b-91b9-e9141bca239b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ca12171-aeba-486b-91b9-e9141bca239b" (UID: "3ca12171-aeba-486b-91b9-e9141bca239b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:20:44.083902 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.083883    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca12171-aeba-486b-91b9-e9141bca239b-kube-api-access-xs5v7" (OuterVolumeSpecName: "kube-api-access-xs5v7") pod "3ca12171-aeba-486b-91b9-e9141bca239b" (UID: "3ca12171-aeba-486b-91b9-e9141bca239b"). InnerVolumeSpecName "kube-api-access-xs5v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:20:44.083963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.083910    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-kube-api-access-4dpgc" (OuterVolumeSpecName: "kube-api-access-4dpgc") pod "8d87eb1f-02bb-45b2-9c54-6144aebe2e28" (UID: "8d87eb1f-02bb-45b2-9c54-6144aebe2e28"). InnerVolumeSpecName "kube-api-access-4dpgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:20:44.154323 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.154229    2561 generic.go:356] "Generic (PLEG): container finished" podID="8d87eb1f-02bb-45b2-9c54-6144aebe2e28" containerID="bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03" exitCode=0
Jul 31 22:20:44.154323 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.154275    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"
Jul 31 22:20:44.154323 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.154313    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz" event={"ID":"8d87eb1f-02bb-45b2-9c54-6144aebe2e28","Type":"ContainerDied","Data":"bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03"}
Jul 31 22:20:44.154604 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.154370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz" event={"ID":"8d87eb1f-02bb-45b2-9c54-6144aebe2e28","Type":"ContainerDied","Data":"2fc21b35c5729c59e720a0e8fd93dfe75686ece00956d053c985d5a23b2f0dad"}
Jul 31 22:20:44.154604 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.154395    2561 scope.go:117] "RemoveContainer" containerID="bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03"
Jul 31 22:20:44.155778 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.155755    2561 generic.go:356] "Generic (PLEG): container finished" podID="3ca12171-aeba-486b-91b9-e9141bca239b" containerID="157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d" exitCode=0
Jul 31 22:20:44.155911 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.155812    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"
Jul 31 22:20:44.155911 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.155852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4" event={"ID":"3ca12171-aeba-486b-91b9-e9141bca239b","Type":"ContainerDied","Data":"157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d"}
Jul 31 22:20:44.155911 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.155901    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4" event={"ID":"3ca12171-aeba-486b-91b9-e9141bca239b","Type":"ContainerDied","Data":"90af78143dc8b1072ebed09c1009cf361ca286df62ea92623312866b3395b154"}
Jul 31 22:20:44.157376 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.157348    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ddaf908-ad53-4b1b-aa10-c3d3f582860c" containerID="daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0" exitCode=0
Jul 31 22:20:44.157497 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.157392    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"
Jul 31 22:20:44.157497 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.157405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb" event={"ID":"7ddaf908-ad53-4b1b-aa10-c3d3f582860c","Type":"ContainerDied","Data":"daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0"}
Jul 31 22:20:44.157497 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.157446    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb" event={"ID":"7ddaf908-ad53-4b1b-aa10-c3d3f582860c","Type":"ContainerDied","Data":"fd99af72ae9011aa280a660c8fc8810f5aa348e346bbf17d2cde6ec8e9def46d"}
Jul 31 22:20:44.164054 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.164027    2561 scope.go:117] "RemoveContainer" containerID="bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03"
Jul 31 22:20:44.164459 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:20:44.164435    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03\": container with ID starting with bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03 not found: ID does not exist" containerID="bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03"
Jul 31 22:20:44.164562 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.164471    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03"} err="failed to get container status \"bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03\": rpc error: code = NotFound desc = could not find container \"bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03\": container with ID starting with bf2159a13c7f180718c68bfa2d2f8f53bcdff4fccaa175b2c74efc89959a7b03 not found: ID does not exist"
Jul 31 22:20:44.164562 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.164499    2561 scope.go:117] "RemoveContainer" containerID="157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d"
Jul 31 22:20:44.174153 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.174118    2561 scope.go:117] "RemoveContainer" containerID="157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d"
Jul 31 22:20:44.174439 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:20:44.174421    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d\": container with ID starting with 157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d not found: ID does not exist" containerID="157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d"
Jul 31 22:20:44.174497 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.174447    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d"} err="failed to get container status \"157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d\": rpc error: code = NotFound desc = could not find container \"157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d\": container with ID starting with 157e8a883e6b426cf91501e22531dda78377e6d21c021d319cfc95d07805145d not found: ID does not exist"
Jul 31 22:20:44.174497 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.174468    2561 scope.go:117] "RemoveContainer" containerID="daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0"
Jul 31 22:20:44.179124 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.179100    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"]
Jul 31 22:20:44.182869 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.182846    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs5v7\" (UniqueName: \"kubernetes.io/projected/3ca12171-aeba-486b-91b9-e9141bca239b-kube-api-access-xs5v7\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:44.182869 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.182869    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:44.183014 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.182878    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dpgc\" (UniqueName: \"kubernetes.io/projected/8d87eb1f-02bb-45b2-9c54-6144aebe2e28-kube-api-access-4dpgc\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:44.183014 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.182888    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca12171-aeba-486b-91b9-e9141bca239b-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:20:44.183904 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.183881    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d193d8-59cbdf9cb4-wfkhz"]
Jul 31 22:20:44.183961 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.183915    2561 scope.go:117] "RemoveContainer" containerID="daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0"
Jul 31 22:20:44.184243 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:20:44.184224    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0\": container with ID starting with daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0 not found: ID does not exist" containerID="daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0"
Jul 31 22:20:44.184299 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.184253    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0"} err="failed to get container status \"daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0\": rpc error: code = NotFound desc = could not find container \"daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0\": container with ID starting with daac1662c8046148e0fe473d674ef108a8d8a99ba730e54f7d2e5087f940ffa0 not found: ID does not exist"
Jul 31 22:20:44.198159 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.195738    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"]
Jul 31 22:20:44.202589 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.202555    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d193d8-58d78b9b94-9fzqb"]
Jul 31 22:20:44.211733 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.211700    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"]
Jul 31 22:20:44.215070 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.215037    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d193d8-69bbd4cb48-mv7w4"]
Jul 31 22:20:44.227819 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.227781    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca12171-aeba-486b-91b9-e9141bca239b" path="/var/lib/kubelet/pods/3ca12171-aeba-486b-91b9-e9141bca239b/volumes"
Jul 31 22:20:44.228212 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.228198    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ddaf908-ad53-4b1b-aa10-c3d3f582860c" path="/var/lib/kubelet/pods/7ddaf908-ad53-4b1b-aa10-c3d3f582860c/volumes"
Jul 31 22:20:44.228519 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:44.228507    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d87eb1f-02bb-45b2-9c54-6144aebe2e28" path="/var/lib/kubelet/pods/8d87eb1f-02bb-45b2-9c54-6144aebe2e28/volumes"
Jul 31 22:20:50.949302 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949267    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"]
Jul 31 22:20:50.949799 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949775    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b66ad9f1-05a0-4ae4-aea5-45d6832af67b" containerName="payload-processing"
Jul 31 22:20:50.949799 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949799    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b66ad9f1-05a0-4ae4-aea5-45d6832af67b" containerName="payload-processing"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949812    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21b84d5-ad18-407f-8775-05e01f04762e" containerName="payload-pre-processing"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949821    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21b84d5-ad18-407f-8775-05e01f04762e" containerName="payload-pre-processing"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949839    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949847    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949862    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ddaf908-ad53-4b1b-aa10-c3d3f582860c" containerName="maas-api"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949870    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ddaf908-ad53-4b1b-aa10-c3d3f582860c" containerName="maas-api"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949884    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca12171-aeba-486b-91b9-e9141bca239b" containerName="payload-pre-processing"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949892    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca12171-aeba-486b-91b9-e9141bca239b" containerName="payload-pre-processing"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949925    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864967bf-1787-42b2-b4f4-0985613b036e" containerName="maas-api"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949933    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="864967bf-1787-42b2-b4f4-0985613b036e" containerName="maas-api"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949946    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d87eb1f-02bb-45b2-9c54-6144aebe2e28" containerName="payload-processing"
Jul 31 22:20:50.950006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.949953    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d87eb1f-02bb-45b2-9c54-6144aebe2e28" containerName="payload-processing"
Jul 31 22:20:50.950608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.950046    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b66ad9f1-05a0-4ae4-aea5-45d6832af67b" containerName="payload-processing"
Jul 31 22:20:50.950608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.950059    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d87eb1f-02bb-45b2-9c54-6144aebe2e28" containerName="payload-processing"
Jul 31 22:20:50.950608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.950070    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ddaf908-ad53-4b1b-aa10-c3d3f582860c" containerName="maas-api"
Jul 31 22:20:50.950608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.950081    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21b84d5-ad18-407f-8775-05e01f04762e" containerName="payload-pre-processing"
Jul 31 22:20:50.950608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.950092    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy"
Jul 31 22:20:50.950608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.950102    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="864967bf-1787-42b2-b4f4-0985613b036e" containerName="maas-api"
Jul 31 22:20:50.950608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.950121    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca12171-aeba-486b-91b9-e9141bca239b" containerName="payload-pre-processing"
Jul 31 22:20:50.950608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.950145    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy"
Jul 31 22:20:50.955018 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.954999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:50.957404 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.957385    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ee9929-openshift-default-dockercfg-f9qsw\""
Jul 31 22:20:50.957656 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.957642    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 22:20:50.964487 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:50.964462    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"]
Jul 31 22:20:51.049845 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.049811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-certs\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.050002 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.049852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istiod-ca-cert\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.050002 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.049886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-envoy\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.050002 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.049957    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-credential-socket\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.050124 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.050026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-data\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.050124 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.050049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-socket\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.050124 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.050066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-token\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.050124 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.050098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9j9f\" (UniqueName: \"kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-kube-api-access-q9j9f\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.050283 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.050163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-podinfo\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151201 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151104    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-data\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151409 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-socket\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151409 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151248    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-token\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151409 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9j9f\" (UniqueName: \"kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-kube-api-access-q9j9f\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151409 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-podinfo\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151409 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-certs\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151648 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istiod-ca-cert\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151648 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-envoy\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151648 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-credential-socket\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151648 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-data\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151854 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-socket\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.151854 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.151798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-credential-socket\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.152274 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.152246    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-certs\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.152378 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.152280    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istiod-ca-cert\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.153690 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.153668    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-envoy\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.154011 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.153990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-podinfo\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.159657 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.159634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9j9f\" (UniqueName: \"kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-kube-api-access-q9j9f\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.159746 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.159638    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-token\") pod \"e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") " pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.266652 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.266556    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:51.406648 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.406617    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"]
Jul 31 22:20:51.409777 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:20:51.409747    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b357f0_1a81_4d91_9ed2_10db5b9f4a29.slice/crio-462bd79286a49431e4b25d30ced0ac5c163f8816c269b8495ef54870961c4836 WatchSource:0}: Error finding container 462bd79286a49431e4b25d30ced0ac5c163f8816c269b8495ef54870961c4836: Status 404 returned error can't find the container with id 462bd79286a49431e4b25d30ced0ac5c163f8816c269b8495ef54870961c4836
Jul 31 22:20:51.412142 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.412094    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:20:51.412206 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.412179    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:20:51.412251 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:51.412207    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:20:52.201868 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:52.201828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" event={"ID":"31b357f0-1a81-4d91-9ed2-10db5b9f4a29","Type":"ContainerStarted","Data":"026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4"}
Jul 31 22:20:52.202276 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:52.201894    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" event={"ID":"31b357f0-1a81-4d91-9ed2-10db5b9f4a29","Type":"ContainerStarted","Data":"462bd79286a49431e4b25d30ced0ac5c163f8816c269b8495ef54870961c4836"}
Jul 31 22:20:52.223882 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:52.223826    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" podStartSLOduration=2.223811762 podStartE2EDuration="2.223811762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:20:52.22208451 +0000 UTC m=+2692.484832887" watchObservedRunningTime="2026-07-31 22:20:52.223811762 +0000 UTC m=+2692.486560125"
Jul 31 22:20:52.267124 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:52.267029    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:52.272151 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:52.272103    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:53.207408 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:53.207372    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:53.208395 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:53.208375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:20:55.373484 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.373441    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"]
Jul 31 22:20:55.374660 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.374627    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy"
Jul 31 22:20:55.374660 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.374660    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="92893c4d-43d6-42e4-a244-d7247e9acc44" containerName="istio-proxy"
Jul 31 22:20:55.378238 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.378215    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:55.380247 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.380222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ee9929-serving-cert\""
Jul 31 22:20:55.388921 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.388830    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"]
Jul 31 22:20:55.495476 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.495438    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e2b429-211b-48b7-9460-0fd73202bef3-maas-api-tls\") pod \"maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:55.495655 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.495484    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e2b429-211b-48b7-9460-0fd73202bef3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:55.495655 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.495588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmhm\" (UniqueName: \"kubernetes.io/projected/f7e2b429-211b-48b7-9460-0fd73202bef3-kube-api-access-npmhm\") pod \"maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:55.596374 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.596331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e2b429-211b-48b7-9460-0fd73202bef3-maas-api-tls\") pod \"maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:55.596374 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.596382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e2b429-211b-48b7-9460-0fd73202bef3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:55.596622 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.596445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmhm\" (UniqueName: \"kubernetes.io/projected/f7e2b429-211b-48b7-9460-0fd73202bef3-kube-api-access-npmhm\") pod \"maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:55.596622 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:20:55.596570    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-serving-cert: secret "maas-api-e2e-shared-a-ee9929-serving-cert" not found
Jul 31 22:20:55.596705 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:20:55.596642    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e2b429-211b-48b7-9460-0fd73202bef3-maas-api-tls podName:f7e2b429-211b-48b7-9460-0fd73202bef3 nodeName:}" failed. No retries permitted until 2026-07-31 22:20:56.096621195 +0000 UTC m=+2696.359369556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7e2b429-211b-48b7-9460-0fd73202bef3-maas-api-tls") pod "maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9" (UID: "f7e2b429-211b-48b7-9460-0fd73202bef3") : secret "maas-api-e2e-shared-a-ee9929-serving-cert" not found
Jul 31 22:20:55.597074 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.597055    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e2b429-211b-48b7-9460-0fd73202bef3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:55.609123 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.609093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmhm\" (UniqueName: \"kubernetes.io/projected/f7e2b429-211b-48b7-9460-0fd73202bef3-kube-api-access-npmhm\") pod \"maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:55.990094 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.990056    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"]
Jul 31 22:20:55.993592 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.993575    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:20:55.996188 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.996159    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-ee9929-dockercfg-ggmz5\""
Jul 31 22:20:55.996344 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:55.996261    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-ee9929\""
Jul 31 22:20:56.002930 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.002906    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"]
Jul 31 22:20:56.101478 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.101432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c3291d-abfd-4f5c-a402-eea21758eafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc\" (UID: \"c4c3291d-abfd-4f5c-a402-eea21758eafc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:20:56.101689 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.101482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87m8d\" (UniqueName: \"kubernetes.io/projected/c4c3291d-abfd-4f5c-a402-eea21758eafc-kube-api-access-87m8d\") pod \"payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc\" (UID: \"c4c3291d-abfd-4f5c-a402-eea21758eafc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:20:56.101689 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.101612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e2b429-211b-48b7-9460-0fd73202bef3-maas-api-tls\") pod \"maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:56.104214 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.104187    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e2b429-211b-48b7-9460-0fd73202bef3-maas-api-tls\") pod \"maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:56.202373 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.202333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c3291d-abfd-4f5c-a402-eea21758eafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc\" (UID: \"c4c3291d-abfd-4f5c-a402-eea21758eafc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:20:56.202373 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.202380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87m8d\" (UniqueName: \"kubernetes.io/projected/c4c3291d-abfd-4f5c-a402-eea21758eafc-kube-api-access-87m8d\") pod \"payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc\" (UID: \"c4c3291d-abfd-4f5c-a402-eea21758eafc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:20:56.203090 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.203064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c3291d-abfd-4f5c-a402-eea21758eafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc\" (UID: \"c4c3291d-abfd-4f5c-a402-eea21758eafc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:20:56.210468 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.210442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87m8d\" (UniqueName: \"kubernetes.io/projected/c4c3291d-abfd-4f5c-a402-eea21758eafc-kube-api-access-87m8d\") pod \"payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc\" (UID: \"c4c3291d-abfd-4f5c-a402-eea21758eafc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:20:56.291770 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.291734    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:56.307221 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.307114    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:20:56.443946 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.443914    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"]
Jul 31 22:20:56.445659 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:20:56.445628    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e2b429_211b_48b7_9460_0fd73202bef3.slice/crio-0c8c474b21335a49efa1b09d3e18e76d3930c1659add247bec4689b85ca8b891 WatchSource:0}: Error finding container 0c8c474b21335a49efa1b09d3e18e76d3930c1659add247bec4689b85ca8b891: Status 404 returned error can't find the container with id 0c8c474b21335a49efa1b09d3e18e76d3930c1659add247bec4689b85ca8b891
Jul 31 22:20:56.467405 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:56.467378    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"]
Jul 31 22:20:56.469677 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:20:56.469647    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c3291d_abfd_4f5c_a402_eea21758eafc.slice/crio-c41717ac07c77e8df9d5e4112bd8edd95176a73258f9f5a4dff0ad64f400cd1f WatchSource:0}: Error finding container c41717ac07c77e8df9d5e4112bd8edd95176a73258f9f5a4dff0ad64f400cd1f: Status 404 returned error can't find the container with id c41717ac07c77e8df9d5e4112bd8edd95176a73258f9f5a4dff0ad64f400cd1f
Jul 31 22:20:57.228518 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:57.228410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc" event={"ID":"c4c3291d-abfd-4f5c-a402-eea21758eafc","Type":"ContainerStarted","Data":"111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a"}
Jul 31 22:20:57.228518 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:57.228481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc" event={"ID":"c4c3291d-abfd-4f5c-a402-eea21758eafc","Type":"ContainerStarted","Data":"c41717ac07c77e8df9d5e4112bd8edd95176a73258f9f5a4dff0ad64f400cd1f"}
Jul 31 22:20:57.228756 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:57.228525    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:20:57.233471 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:57.233438    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9" event={"ID":"f7e2b429-211b-48b7-9460-0fd73202bef3","Type":"ContainerStarted","Data":"5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9"}
Jul 31 22:20:57.233649 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:57.233489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9" event={"ID":"f7e2b429-211b-48b7-9460-0fd73202bef3","Type":"ContainerStarted","Data":"0c8c474b21335a49efa1b09d3e18e76d3930c1659add247bec4689b85ca8b891"}
Jul 31 22:20:57.233649 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:57.233518    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:20:57.243076 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:57.243016    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc" podStartSLOduration=2.242994477 podStartE2EDuration="2.242994477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:20:57.242653027 +0000 UTC m=+2697.505401399" watchObservedRunningTime="2026-07-31 22:20:57.242994477 +0000 UTC m=+2697.505742843"
Jul 31 22:20:57.257649 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:20:57.257585    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9" podStartSLOduration=1.9011881590000002 podStartE2EDuration="2.257564139s" totalImagesPullingTime="356.37598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:20:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:20:57.25700841 +0000 UTC m=+2697.519756775" watchObservedRunningTime="2026-07-31 22:20:57.257564139 +0000 UTC m=+2697.520312503"
Jul 31 22:21:01.022729 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:01.022701    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:21:01.027562 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:01.027532    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:21:03.244317 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:03.244277    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:21:08.208846 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:08.208805    2561 patch_prober.go:28] interesting pod/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Jul 31 22:21:08.209267 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:08.208870    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Jul 31 22:21:08.240025 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:08.239993    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:21:08.291161 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:08.291107    2561 generic.go:356] "Generic (PLEG): container finished" podID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerID="026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4" exitCode=0
Jul 31 22:21:08.291333 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:08.291172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" event={"ID":"31b357f0-1a81-4d91-9ed2-10db5b9f4a29","Type":"ContainerDied","Data":"026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4"}
Jul 31 22:21:08.291612 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:08.291592    2561 scope.go:117] "RemoveContainer" containerID="026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4"
Jul 31 22:21:08.292650 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:08.292618    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:21:08.292760 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:08.292687    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:21:08.292760 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:08.292721    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:21:09.302101 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:09.302051    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" event={"ID":"31b357f0-1a81-4d91-9ed2-10db5b9f4a29","Type":"ContainerStarted","Data":"b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0"}
Jul 31 22:21:10.267450 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:10.267409    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:21:10.267697 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:10.267674    2561 patch_prober.go:28] interesting pod/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Jul 31 22:21:10.267805 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:10.267746    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Jul 31 22:21:11.267884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:11.267844    2561 patch_prober.go:28] interesting pod/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Jul 31 22:21:11.268331 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:11.267915    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Jul 31 22:21:12.267734 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:12.267691    2561 patch_prober.go:28] interesting pod/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Jul 31 22:21:12.267925 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:12.267751    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Jul 31 22:21:13.267357 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:13.267318    2561 patch_prober.go:28] interesting pod/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Jul 31 22:21:13.267571 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:13.267392    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Jul 31 22:21:14.267772 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:14.267730    2561 patch_prober.go:28] interesting pod/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Jul 31 22:21:14.268219 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:14.267802    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Jul 31 22:21:15.270940 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:15.270904    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:21:15.271346 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:15.271248    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:21:15.271964 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:15.271942    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:21:19.892002 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:19.891956    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"]
Jul 31 22:21:19.895720 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:19.895695    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:19.900682 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:19.900654    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-21e950-openshift-default-dockercfg-mrkmt\""
Jul 31 22:21:19.919710 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:19.919672    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"]
Jul 31 22:21:20.029084 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.029046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dh5t\" (UniqueName: \"kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-kube-api-access-6dh5t\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.029343 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.029092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-data\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.029343 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.029166    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-token\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.029343 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.029225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-socket\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.029343 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.029278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-envoy\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.029343 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.029303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istiod-ca-cert\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.029569 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.029389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-credential-socket\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.029569 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.029427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-podinfo\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.029569 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.029448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-certs\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.130632 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.130593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-token\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.130632 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.130630    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-socket\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.130884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.130655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-envoy\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.130884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.130673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istiod-ca-cert\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.130884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.130710    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-credential-socket\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.130884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.130731    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-podinfo\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.130884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.130748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-certs\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.130884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.130798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dh5t\" (UniqueName: \"kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-kube-api-access-6dh5t\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.130884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.130881    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-data\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.131284 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.131208    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-socket\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.131390 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.131362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-credential-socket\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.131464 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.131419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-data\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.131507 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.131466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-certs\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.131620 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.131600    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istiod-ca-cert\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.133035 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.133014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-envoy\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.133263 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.133245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-podinfo\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.139402 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.139376    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-token\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.139583 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.139564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dh5t\" (UniqueName: \"kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-kube-api-access-6dh5t\") pod \"e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") " pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.207546 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.207444    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:20.353159 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.353117    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"]
Jul 31 22:21:20.355399 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:21:20.355361    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd857d9e1_caf3_42a1_b68e_5a6e9427e23b.slice/crio-0cc134f3f3921014b6f81a366dfa48acc7cefce53e719e8dbfe4c3e72afac179 WatchSource:0}: Error finding container 0cc134f3f3921014b6f81a366dfa48acc7cefce53e719e8dbfe4c3e72afac179: Status 404 returned error can't find the container with id 0cc134f3f3921014b6f81a366dfa48acc7cefce53e719e8dbfe4c3e72afac179
Jul 31 22:21:20.357757 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.357722    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:21:20.357876 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.357803    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:21:20.357876 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.357838    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:21:20.364026 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:20.364004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" event={"ID":"d857d9e1-caf3-42a1-b68e-5a6e9427e23b","Type":"ContainerStarted","Data":"0cc134f3f3921014b6f81a366dfa48acc7cefce53e719e8dbfe4c3e72afac179"}
Jul 31 22:21:21.370833 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:21.370792    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" event={"ID":"d857d9e1-caf3-42a1-b68e-5a6e9427e23b","Type":"ContainerStarted","Data":"d36e883372f6f1671fc0150ff66a34fe33b713e373548d2234d8880e9b875366"}
Jul 31 22:21:21.389931 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:21.389875    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" podStartSLOduration=2.38985879 podStartE2EDuration="2.38985879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:21:21.387921803 +0000 UTC m=+2721.650670167" watchObservedRunningTime="2026-07-31 22:21:21.38985879 +0000 UTC m=+2721.652607153"
Jul 31 22:21:22.207678 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:22.207631    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:22.213013 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:22.212982    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:22.375779 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:22.375738    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:22.376781 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:22.376757    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:25.179185 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.179127    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"]
Jul 31 22:21:25.183036 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.183011    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:21:25.185071 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.185044    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-21e950\""
Jul 31 22:21:25.185071 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.185060    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-21e950-dockercfg-xn2hl\""
Jul 31 22:21:25.195957 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.195926    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"]
Jul 31 22:21:25.284251 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.284206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af975e5f-aa5e-4237-9818-1c8a6f134748-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p\" (UID: \"af975e5f-aa5e-4237-9818-1c8a6f134748\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:21:25.284439 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.284418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlsc7\" (UniqueName: \"kubernetes.io/projected/af975e5f-aa5e-4237-9818-1c8a6f134748-kube-api-access-xlsc7\") pod \"payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p\" (UID: \"af975e5f-aa5e-4237-9818-1c8a6f134748\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:21:25.385680 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.385636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlsc7\" (UniqueName: \"kubernetes.io/projected/af975e5f-aa5e-4237-9818-1c8a6f134748-kube-api-access-xlsc7\") pod \"payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p\" (UID: \"af975e5f-aa5e-4237-9818-1c8a6f134748\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:21:25.385899 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.385721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af975e5f-aa5e-4237-9818-1c8a6f134748-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p\" (UID: \"af975e5f-aa5e-4237-9818-1c8a6f134748\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:21:25.386468 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.386447    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af975e5f-aa5e-4237-9818-1c8a6f134748-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p\" (UID: \"af975e5f-aa5e-4237-9818-1c8a6f134748\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:21:25.394860 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.394832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlsc7\" (UniqueName: \"kubernetes.io/projected/af975e5f-aa5e-4237-9818-1c8a6f134748-kube-api-access-xlsc7\") pod \"payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p\" (UID: \"af975e5f-aa5e-4237-9818-1c8a6f134748\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:21:25.497236 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.497115    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:21:25.640045 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:25.640018    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"]
Jul 31 22:21:25.642610 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:21:25.642571    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf975e5f_aa5e_4237_9818_1c8a6f134748.slice/crio-57d8d25c362991826946dd33e693450e59d42a6a1539934ab58025cd4f6c5bba WatchSource:0}: Error finding container 57d8d25c362991826946dd33e693450e59d42a6a1539934ab58025cd4f6c5bba: Status 404 returned error can't find the container with id 57d8d25c362991826946dd33e693450e59d42a6a1539934ab58025cd4f6c5bba
Jul 31 22:21:26.399321 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:26.399283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p" event={"ID":"af975e5f-aa5e-4237-9818-1c8a6f134748","Type":"ContainerStarted","Data":"3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d"}
Jul 31 22:21:26.399814 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:26.399343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p" event={"ID":"af975e5f-aa5e-4237-9818-1c8a6f134748","Type":"ContainerStarted","Data":"57d8d25c362991826946dd33e693450e59d42a6a1539934ab58025cd4f6c5bba"}
Jul 31 22:21:26.399814 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:26.399473    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:21:26.414212 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:26.414118    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p" podStartSLOduration=1.414097435 podStartE2EDuration="1.414097435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:21:26.412279512 +0000 UTC m=+2726.675027869" watchObservedRunningTime="2026-07-31 22:21:26.414097435 +0000 UTC m=+2726.676845800"
Jul 31 22:21:37.378057 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:37.378013    2561 patch_prober.go:28] interesting pod/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 31 22:21:37.378527 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:37.378078    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 31 22:21:37.407712 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:37.407622    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:21:37.460388 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:37.460293    2561 generic.go:356] "Generic (PLEG): container finished" podID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerID="d36e883372f6f1671fc0150ff66a34fe33b713e373548d2234d8880e9b875366" exitCode=0
Jul 31 22:21:37.460388 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:37.460360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" event={"ID":"d857d9e1-caf3-42a1-b68e-5a6e9427e23b","Type":"ContainerDied","Data":"d36e883372f6f1671fc0150ff66a34fe33b713e373548d2234d8880e9b875366"}
Jul 31 22:21:37.460972 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:37.460953    2561 scope.go:117] "RemoveContainer" containerID="d36e883372f6f1671fc0150ff66a34fe33b713e373548d2234d8880e9b875366"
Jul 31 22:21:37.462223 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:37.462180    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:21:37.462319 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:37.462260    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:21:37.462319 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:37.462294    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:21:38.467182 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:38.467120    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" event={"ID":"d857d9e1-caf3-42a1-b68e-5a6e9427e23b","Type":"ContainerStarted","Data":"4bf2419d7407498b356def4006f6b81d2f2446bcedc06838224b0ac7dece8aa0"}
Jul 31 22:21:39.208689 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:39.208648    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:39.209210 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:39.209020    2561 patch_prober.go:28] interesting pod/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 31 22:21:39.209423 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:39.209392    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 31 22:21:40.208333 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:40.208289    2561 patch_prober.go:28] interesting pod/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 31 22:21:40.208971 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:40.208933    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 31 22:21:41.208269 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:41.208217    2561 patch_prober.go:28] interesting pod/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 31 22:21:41.208563 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:41.208340    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 31 22:21:42.208343 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:42.208299    2561 patch_prober.go:28] interesting pod/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 31 22:21:42.208707 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:42.208675    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 31 22:21:43.208117 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:43.208074    2561 patch_prober.go:28] interesting pod/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 31 22:21:43.208458 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:43.208424    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 31 22:21:44.212654 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:44.212622    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:44.213083 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:44.212908    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:21:44.213721 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:21:44.213698    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:25:37.227267 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.227171    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"]
Jul 31 22:25:37.233219 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.231748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.236727 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.236693    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ee9929\"/\"default-dockercfg-px6qf\""
Jul 31 22:25:37.237186 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.236706    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ee9929\"/\"kube-root-ca.crt\""
Jul 31 22:25:37.237186 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.236712    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ee9929\"/\"openshift-service-ca.crt\""
Jul 31 22:25:37.237186 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.237128    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ee9929\"/\"sub-test-model-ee9929-kserve-self-signed-certs\""
Jul 31 22:25:37.240608 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.240579    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"]
Jul 31 22:25:37.336744 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.336699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-home\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.336744 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.336739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tls-certs\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.336994 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.336778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbr6p\" (UniqueName: \"kubernetes.io/projected/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-kube-api-access-jbr6p\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.336994 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.336829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-dshm\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.336994 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.336852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-model-cache\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.336994 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.336874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tmp-dir\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.438106 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.438058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbr6p\" (UniqueName: \"kubernetes.io/projected/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-kube-api-access-jbr6p\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.438350 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.438176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-dshm\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.438350 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.438212    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-model-cache\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.438350 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.438249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tmp-dir\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.438350 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.438340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-home\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.438517 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.438363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tls-certs\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.438689 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.438667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-model-cache\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.438756 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.438719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tmp-dir\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.438810 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.438759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-home\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.440691 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.440672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-dshm\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.440793 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.440777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tls-certs\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.446666 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.446639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbr6p\" (UniqueName: \"kubernetes.io/projected/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-kube-api-access-jbr6p\") pod \"sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") " pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.546159 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.546079    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:37.693309 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:37.693247    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"]
Jul 31 22:25:37.695973 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:25:37.695942    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcac30b7e_6c06_4404_aa9c_f0ed501c8bf5.slice/crio-5ada8a85d3dce72f3f12c2239503147de38a5327242c1c49a59c492e091e4010 WatchSource:0}: Error finding container 5ada8a85d3dce72f3f12c2239503147de38a5327242c1c49a59c492e091e4010: Status 404 returned error can't find the container with id 5ada8a85d3dce72f3f12c2239503147de38a5327242c1c49a59c492e091e4010
Jul 31 22:25:38.695198 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:38.695163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42" event={"ID":"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5","Type":"ContainerStarted","Data":"b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d"}
Jul 31 22:25:38.695198 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:38.695206    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42" event={"ID":"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5","Type":"ContainerStarted","Data":"5ada8a85d3dce72f3f12c2239503147de38a5327242c1c49a59c492e091e4010"}
Jul 31 22:25:38.729075 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:38.729021    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42" podStartSLOduration=1.729003097 podStartE2EDuration="1.729003097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:25:38.72772082 +0000 UTC m=+2978.990469187" watchObservedRunningTime="2026-07-31 22:25:38.729003097 +0000 UTC m=+2978.991751528"
Jul 31 22:25:47.546271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:47.546225    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:47.546271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:47.546282    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:47.559214 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:47.559185    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:25:47.753725 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:25:47.753693    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:26:01.052523 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:26:01.052492    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:26:01.057662 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:26:01.057640    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:26:01.166690 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:26:01.166655    2561 scope.go:117] "RemoveContainer" containerID="459fcc5637ded75aebdf001ad6ddb7eed9c63e808e7706c2cea2ce18ec700737"
Jul 31 22:27:30.211285 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:30.211247    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"]
Jul 31 22:27:45.234226 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.234181    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42" podUID="cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" containerName="main" containerID="cri-o://b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d" gracePeriod=45
Jul 31 22:27:45.484580 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.484513    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:27:45.494111 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.494078    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbr6p\" (UniqueName: \"kubernetes.io/projected/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-kube-api-access-jbr6p\") pod \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") "
Jul 31 22:27:45.494314 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.494295    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tls-certs\") pod \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") "
Jul 31 22:27:45.494372 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.494344    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-home\") pod \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") "
Jul 31 22:27:45.494416 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.494381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tmp-dir\") pod \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") "
Jul 31 22:27:45.494416 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.494409    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-dshm\") pod \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") "
Jul 31 22:27:45.494520 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.494463    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-model-cache\") pod \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\" (UID: \"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5\") "
Jul 31 22:27:45.494825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.494794    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-home" (OuterVolumeSpecName: "home") pod "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" (UID: "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:27:45.494923 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.494825    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" (UID: "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:27:45.494987 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.494910    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-home\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:27:45.494987 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.494956    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tmp-dir\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:27:45.495089 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.494974    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-model-cache" (OuterVolumeSpecName: "model-cache") pod "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" (UID: "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:27:45.496871 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.496846    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" (UID: "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:27:45.497029 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.497013    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-dshm" (OuterVolumeSpecName: "dshm") pod "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" (UID: "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:27:45.497199 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.497182    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-kube-api-access-jbr6p" (OuterVolumeSpecName: "kube-api-access-jbr6p") pod "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" (UID: "cac30b7e-6c06-4404-aa9c-f0ed501c8bf5"). InnerVolumeSpecName "kube-api-access-jbr6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:27:45.595809 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.595764    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-tls-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:27:45.595809 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.595801    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-dshm\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:27:45.595809 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.595810    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-model-cache\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:27:45.595809 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:45.595821    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbr6p\" (UniqueName: \"kubernetes.io/projected/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5-kube-api-access-jbr6p\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:27:46.335522 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:46.335480    2561 generic.go:356] "Generic (PLEG): container finished" podID="cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" containerID="b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d" exitCode=0
Jul 31 22:27:46.336001 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:46.335539    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"
Jul 31 22:27:46.336001 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:46.335550    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42" event={"ID":"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5","Type":"ContainerDied","Data":"b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d"}
Jul 31 22:27:46.336001 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:46.335589    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42" event={"ID":"cac30b7e-6c06-4404-aa9c-f0ed501c8bf5","Type":"ContainerDied","Data":"5ada8a85d3dce72f3f12c2239503147de38a5327242c1c49a59c492e091e4010"}
Jul 31 22:27:46.336001 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:46.335612    2561 scope.go:117] "RemoveContainer" containerID="b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d"
Jul 31 22:27:46.344900 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:46.344879    2561 scope.go:117] "RemoveContainer" containerID="b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d"
Jul 31 22:27:46.345248 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:27:46.345226    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d\": container with ID starting with b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d not found: ID does not exist" containerID="b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d"
Jul 31 22:27:46.345302 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:46.345258    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d"} err="failed to get container status \"b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d\": rpc error: code = NotFound desc = could not find container \"b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d\": container with ID starting with b26837571d9dbcdcdd994dd70c5ad7c5a3b1effa571c27bee2b8e4ec9425c57d not found: ID does not exist"
Jul 31 22:27:46.352792 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:46.352769    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"]
Jul 31 22:27:46.356803 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:46.356780    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ee9929/sub-test-model-ee9929-kserve-59bdbc6f4f-g7c42"]
Jul 31 22:27:48.227714 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:27:48.227679    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" path="/var/lib/kubelet/pods/cac30b7e-6c06-4404-aa9c-f0ed501c8bf5/volumes"
Jul 31 22:29:36.336810 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.336775    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7527-140539711946016"]
Jul 31 22:29:36.337315 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.337156    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" containerName="main"
Jul 31 22:29:36.337315 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.337169    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" containerName="main"
Jul 31 22:29:36.337315 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.337245    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac30b7e-6c06-4404-aa9c-f0ed501c8bf5" containerName="main"
Jul 31 22:29:36.340382 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.340365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:36.343744 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.343718    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kc9s9\""
Jul 31 22:29:36.353433 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.353408    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7527-140539711946016"]
Jul 31 22:29:36.453056 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.453023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmf7\" (UniqueName: \"kubernetes.io/projected/eb797a58-749c-45d3-84a3-7137fccaa950-kube-api-access-qpmf7\") pod \"test-curl-7527-140539711946016\" (UID: \"eb797a58-749c-45d3-84a3-7137fccaa950\") " pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:36.554415 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.554349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmf7\" (UniqueName: \"kubernetes.io/projected/eb797a58-749c-45d3-84a3-7137fccaa950-kube-api-access-qpmf7\") pod \"test-curl-7527-140539711946016\" (UID: \"eb797a58-749c-45d3-84a3-7137fccaa950\") " pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:36.562849 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.562823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmf7\" (UniqueName: \"kubernetes.io/projected/eb797a58-749c-45d3-84a3-7137fccaa950-kube-api-access-qpmf7\") pod \"test-curl-7527-140539711946016\" (UID: \"eb797a58-749c-45d3-84a3-7137fccaa950\") " pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:36.649841 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.649744    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:36.785373 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.785347    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7527-140539711946016"]
Jul 31 22:29:36.787485 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:29:36.787457    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb797a58_749c_45d3_84a3_7137fccaa950.slice/crio-e39ef9c975ff85cfcb92b9189f907e7388e1a4abca0024100cabf5a3e863c3fb WatchSource:0}: Error finding container e39ef9c975ff85cfcb92b9189f907e7388e1a4abca0024100cabf5a3e863c3fb: Status 404 returned error can't find the container with id e39ef9c975ff85cfcb92b9189f907e7388e1a4abca0024100cabf5a3e863c3fb
Jul 31 22:29:36.788875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.788857    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 22:29:36.888825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:36.888792    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7527-140539711946016" event={"ID":"eb797a58-749c-45d3-84a3-7137fccaa950","Type":"ContainerStarted","Data":"e39ef9c975ff85cfcb92b9189f907e7388e1a4abca0024100cabf5a3e863c3fb"}
Jul 31 22:29:37.894829 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:37.894792    2561 generic.go:356] "Generic (PLEG): container finished" podID="eb797a58-749c-45d3-84a3-7137fccaa950" containerID="7ef5839fc6180f7213f5449c2e07e14f19b96c76f1f909741a0097149fa5e5de" exitCode=0
Jul 31 22:29:37.895257 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:37.894881    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7527-140539711946016" event={"ID":"eb797a58-749c-45d3-84a3-7137fccaa950","Type":"ContainerDied","Data":"7ef5839fc6180f7213f5449c2e07e14f19b96c76f1f909741a0097149fa5e5de"}
Jul 31 22:29:39.036698 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.036668    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:39.078912 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.078883    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7527-140539711946016_eb797a58-749c-45d3-84a3-7137fccaa950/test-curl-7527-140539711946016/0.log"
Jul 31 22:29:39.107715 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.107682    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7527-140539711946016"]
Jul 31 22:29:39.114710 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.114688    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7527-140539711946016"]
Jul 31 22:29:39.179915 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.179834    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpmf7\" (UniqueName: \"kubernetes.io/projected/eb797a58-749c-45d3-84a3-7137fccaa950-kube-api-access-qpmf7\") pod \"eb797a58-749c-45d3-84a3-7137fccaa950\" (UID: \"eb797a58-749c-45d3-84a3-7137fccaa950\") "
Jul 31 22:29:39.181976 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.181944    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb797a58-749c-45d3-84a3-7137fccaa950-kube-api-access-qpmf7" (OuterVolumeSpecName: "kube-api-access-qpmf7") pod "eb797a58-749c-45d3-84a3-7137fccaa950" (UID: "eb797a58-749c-45d3-84a3-7137fccaa950"). InnerVolumeSpecName "kube-api-access-qpmf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:29:39.246683 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.246646    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7527-140539711946016"]
Jul 31 22:29:39.247095 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.247080    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb797a58-749c-45d3-84a3-7137fccaa950" containerName="test-curl-7527-140539711946016"
Jul 31 22:29:39.247185 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.247098    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb797a58-749c-45d3-84a3-7137fccaa950" containerName="test-curl-7527-140539711946016"
Jul 31 22:29:39.247226 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.247203    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb797a58-749c-45d3-84a3-7137fccaa950" containerName="test-curl-7527-140539711946016"
Jul 31 22:29:39.250324 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.250309    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:39.257072 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.257042    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7527-140539711946016"]
Jul 31 22:29:39.281009 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.280984    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpmf7\" (UniqueName: \"kubernetes.io/projected/eb797a58-749c-45d3-84a3-7137fccaa950-kube-api-access-qpmf7\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:29:39.382006 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.381969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q97cz\" (UniqueName: \"kubernetes.io/projected/18e5cbfc-ca34-4d99-8585-9cb4879a6584-kube-api-access-q97cz\") pod \"test-curl-7527-140539711946016\" (UID: \"18e5cbfc-ca34-4d99-8585-9cb4879a6584\") " pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:39.483217 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.483125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q97cz\" (UniqueName: \"kubernetes.io/projected/18e5cbfc-ca34-4d99-8585-9cb4879a6584-kube-api-access-q97cz\") pod \"test-curl-7527-140539711946016\" (UID: \"18e5cbfc-ca34-4d99-8585-9cb4879a6584\") " pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:39.491380 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.491360    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q97cz\" (UniqueName: \"kubernetes.io/projected/18e5cbfc-ca34-4d99-8585-9cb4879a6584-kube-api-access-q97cz\") pod \"test-curl-7527-140539711946016\" (UID: \"18e5cbfc-ca34-4d99-8585-9cb4879a6584\") " pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:39.561575 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.561544    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:39.695046 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.695019    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7527-140539711946016"]
Jul 31 22:29:39.697497 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:29:39.697468    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18e5cbfc_ca34_4d99_8585_9cb4879a6584.slice/crio-3da79f0e6f722b77462432ca12f3fff11bb114078f9dc7deaba9ddf6fceaaf87 WatchSource:0}: Error finding container 3da79f0e6f722b77462432ca12f3fff11bb114078f9dc7deaba9ddf6fceaaf87: Status 404 returned error can't find the container with id 3da79f0e6f722b77462432ca12f3fff11bb114078f9dc7deaba9ddf6fceaaf87
Jul 31 22:29:39.907109 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.907069    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:39.907109 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.907097    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e39ef9c975ff85cfcb92b9189f907e7388e1a4abca0024100cabf5a3e863c3fb"
Jul 31 22:29:39.908381 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.908350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7527-140539711946016" event={"ID":"18e5cbfc-ca34-4d99-8585-9cb4879a6584","Type":"ContainerStarted","Data":"3da79f0e6f722b77462432ca12f3fff11bb114078f9dc7deaba9ddf6fceaaf87"}
Jul 31 22:29:39.918762 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:39.918725    2561 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7527-140539711946016" oldPodUID="eb797a58-749c-45d3-84a3-7137fccaa950" podUID="18e5cbfc-ca34-4d99-8585-9cb4879a6584"
Jul 31 22:29:40.233508 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:40.233472    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb797a58-749c-45d3-84a3-7137fccaa950" path="/var/lib/kubelet/pods/eb797a58-749c-45d3-84a3-7137fccaa950/volumes"
Jul 31 22:29:40.915676 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:40.915642    2561 generic.go:356] "Generic (PLEG): container finished" podID="18e5cbfc-ca34-4d99-8585-9cb4879a6584" containerID="60751edfaa39e07ec8da6a5b22df392a5bb92b62dea10edd82b57c47d887cbfc" exitCode=0
Jul 31 22:29:40.915845 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:40.915722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7527-140539711946016" event={"ID":"18e5cbfc-ca34-4d99-8585-9cb4879a6584","Type":"ContainerDied","Data":"60751edfaa39e07ec8da6a5b22df392a5bb92b62dea10edd82b57c47d887cbfc"}
Jul 31 22:29:42.060498 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:42.060470    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:42.081821 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:42.081794    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7527-140539711946016_18e5cbfc-ca34-4d99-8585-9cb4879a6584/test-curl-7527-140539711946016/0.log"
Jul 31 22:29:42.108805 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:42.108778    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7527-140539711946016"]
Jul 31 22:29:42.114684 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:42.114661    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7527-140539711946016"]
Jul 31 22:29:42.207528 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:42.207439    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q97cz\" (UniqueName: \"kubernetes.io/projected/18e5cbfc-ca34-4d99-8585-9cb4879a6584-kube-api-access-q97cz\") pod \"18e5cbfc-ca34-4d99-8585-9cb4879a6584\" (UID: \"18e5cbfc-ca34-4d99-8585-9cb4879a6584\") "
Jul 31 22:29:42.209616 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:42.209590    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e5cbfc-ca34-4d99-8585-9cb4879a6584-kube-api-access-q97cz" (OuterVolumeSpecName: "kube-api-access-q97cz") pod "18e5cbfc-ca34-4d99-8585-9cb4879a6584" (UID: "18e5cbfc-ca34-4d99-8585-9cb4879a6584"). InnerVolumeSpecName "kube-api-access-q97cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:29:42.228449 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:42.228417    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18e5cbfc-ca34-4d99-8585-9cb4879a6584" path="/var/lib/kubelet/pods/18e5cbfc-ca34-4d99-8585-9cb4879a6584/volumes"
Jul 31 22:29:42.308972 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:42.308944    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q97cz\" (UniqueName: \"kubernetes.io/projected/18e5cbfc-ca34-4d99-8585-9cb4879a6584-kube-api-access-q97cz\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:29:42.927045 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:42.927011    2561 scope.go:117] "RemoveContainer" containerID="60751edfaa39e07ec8da6a5b22df392a5bb92b62dea10edd82b57c47d887cbfc"
Jul 31 22:29:42.927244 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:42.927021    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7527-140539711946016"
Jul 31 22:29:49.044742 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.044708    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"]
Jul 31 22:29:49.045344 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.045327    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e5cbfc-ca34-4d99-8585-9cb4879a6584" containerName="test-curl-7527-140539711946016"
Jul 31 22:29:49.045399 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.045347    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e5cbfc-ca34-4d99-8585-9cb4879a6584" containerName="test-curl-7527-140539711946016"
Jul 31 22:29:49.045464 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.045453    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e5cbfc-ca34-4d99-8585-9cb4879a6584" containerName="test-curl-7527-140539711946016"
Jul 31 22:29:49.050110 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.050088    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.052684 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.052666    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-068335-openshift-default-dockercfg-vtnlh\""
Jul 31 22:29:49.060465 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.060441    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"]
Jul 31 22:29:49.069352 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.069329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-envoy\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.069454 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.069381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-credential-socket\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.069495 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.069453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-podinfo\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.069536 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.069495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-data\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.069571 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.069535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-socket\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.069619 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.069599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-certs\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.069656 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.069640    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-token\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.069690 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.069665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngbt2\" (UniqueName: \"kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-kube-api-access-ngbt2\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.069729 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.069706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istiod-ca-cert\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171078 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171041    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-podinfo\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171317 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-data\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171317 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-socket\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171317 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-certs\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171486 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-token\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171486 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngbt2\" (UniqueName: \"kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-kube-api-access-ngbt2\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171486 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171392    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istiod-ca-cert\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171486 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-envoy\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171688 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-credential-socket\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171688 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-data\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171688 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171556    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-socket\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.171688 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-certs\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.172002 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.171887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-credential-socket\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.172085 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.172065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istiod-ca-cert\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.173644 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.173626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-envoy\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.173985 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.173964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-podinfo\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.180725 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.180696    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngbt2\" (UniqueName: \"kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-kube-api-access-ngbt2\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.180725 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.180712    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-token\") pod \"e2e-ipp-a-068335-openshift-default-759456756d-rvspz\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") " pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.361698 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.361609    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:49.504017 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.503993    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"]
Jul 31 22:29:49.506031 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:29:49.506000    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527acc4e_c0bb_4f02_9bf0_3bd71e5598ad.slice/crio-b5348ce2f04be4a8734c2d025ebc48ac940245376c6a66697f32823a1f0755dd WatchSource:0}: Error finding container b5348ce2f04be4a8734c2d025ebc48ac940245376c6a66697f32823a1f0755dd: Status 404 returned error can't find the container with id b5348ce2f04be4a8734c2d025ebc48ac940245376c6a66697f32823a1f0755dd
Jul 31 22:29:49.508237 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.508203    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:29:49.508316 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.508267    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:29:49.508316 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.508295    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:29:49.964690 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.964654    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" event={"ID":"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad","Type":"ContainerStarted","Data":"46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048"}
Jul 31 22:29:49.964875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.964711    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" event={"ID":"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad","Type":"ContainerStarted","Data":"b5348ce2f04be4a8734c2d025ebc48ac940245376c6a66697f32823a1f0755dd"}
Jul 31 22:29:49.987465 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:49.987409    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" podStartSLOduration=0.98739301 podStartE2EDuration="987.39301ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:29:49.985587854 +0000 UTC m=+3230.248336217" watchObservedRunningTime="2026-07-31 22:29:49.98739301 +0000 UTC m=+3230.250141375"
Jul 31 22:29:50.362161 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:50.362100    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:50.367848 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:50.367822    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:50.970207 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:50.970174    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:50.971087 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:50.971067    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:29:53.703545 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:53.703457    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"]
Jul 31 22:29:53.707099 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:53.707070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:29:53.709480 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:53.709457    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-068335-dockercfg-qptk2\""
Jul 31 22:29:53.709629 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:53.709612    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-068335\""
Jul 31 22:29:53.724492 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:53.724466    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"]
Jul 31 22:29:53.817158 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:53.817108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxfms\" (UniqueName: \"kubernetes.io/projected/221dd229-4d95-4c4e-996d-bc147db7af8b-kube-api-access-fxfms\") pod \"payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r\" (UID: \"221dd229-4d95-4c4e-996d-bc147db7af8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:29:53.817358 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:53.817170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221dd229-4d95-4c4e-996d-bc147db7af8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r\" (UID: \"221dd229-4d95-4c4e-996d-bc147db7af8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:29:53.918052 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:53.918018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxfms\" (UniqueName: \"kubernetes.io/projected/221dd229-4d95-4c4e-996d-bc147db7af8b-kube-api-access-fxfms\") pod \"payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r\" (UID: \"221dd229-4d95-4c4e-996d-bc147db7af8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:29:53.918247 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:53.918069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221dd229-4d95-4c4e-996d-bc147db7af8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r\" (UID: \"221dd229-4d95-4c4e-996d-bc147db7af8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:29:53.918817 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:53.918792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221dd229-4d95-4c4e-996d-bc147db7af8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r\" (UID: \"221dd229-4d95-4c4e-996d-bc147db7af8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:29:53.926271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:53.926250    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxfms\" (UniqueName: \"kubernetes.io/projected/221dd229-4d95-4c4e-996d-bc147db7af8b-kube-api-access-fxfms\") pod \"payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r\" (UID: \"221dd229-4d95-4c4e-996d-bc147db7af8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:29:54.020379 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:54.020300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:29:54.370085 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:54.370049    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"]
Jul 31 22:29:54.373707 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:29:54.373677    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221dd229_4d95_4c4e_996d_bc147db7af8b.slice/crio-3e01cda2065de24b9e4946eee2232e948a52fc30c73b2f22121807f27a386d53 WatchSource:0}: Error finding container 3e01cda2065de24b9e4946eee2232e948a52fc30c73b2f22121807f27a386d53: Status 404 returned error can't find the container with id 3e01cda2065de24b9e4946eee2232e948a52fc30c73b2f22121807f27a386d53
Jul 31 22:29:54.991901 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:54.991862    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r" event={"ID":"221dd229-4d95-4c4e-996d-bc147db7af8b","Type":"ContainerStarted","Data":"ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601"}
Jul 31 22:29:54.992350 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:54.991915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r" event={"ID":"221dd229-4d95-4c4e-996d-bc147db7af8b","Type":"ContainerStarted","Data":"3e01cda2065de24b9e4946eee2232e948a52fc30c73b2f22121807f27a386d53"}
Jul 31 22:29:54.992350 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:54.992013    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:29:55.008591 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:29:55.008533    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r" podStartSLOduration=2.008514608 podStartE2EDuration="2.008514608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:29:55.006248961 +0000 UTC m=+3235.268997326" watchObservedRunningTime="2026-07-31 22:29:55.008514608 +0000 UTC m=+3235.271262974"
Jul 31 22:30:06.005791 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:06.005759    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:30:06.303151 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:06.302250    2561 patch_prober.go:28] interesting pod/e2e-ipp-a-068335-openshift-default-759456756d-rvspz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": read tcp 10.132.0.2:40870->10.132.0.80:15021: read: connection reset by peer" start-of-body=
Jul 31 22:30:06.303151 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:06.302318    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": read tcp 10.132.0.2:40870->10.132.0.80:15021: read: connection reset by peer"
Jul 31 22:30:07.063245 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:07.063209    2561 generic.go:356] "Generic (PLEG): container finished" podID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerID="46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048" exitCode=0
Jul 31 22:30:07.063730 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:07.063288    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" event={"ID":"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad","Type":"ContainerDied","Data":"46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048"}
Jul 31 22:30:07.063730 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:07.063680    2561 scope.go:117] "RemoveContainer" containerID="46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048"
Jul 31 22:30:07.068780 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:07.065033    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:30:07.068780 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:07.065259    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:30:07.068780 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:07.065360    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:30:08.070190 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:08.070156    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" event={"ID":"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad","Type":"ContainerStarted","Data":"5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8"}
Jul 31 22:30:08.362853 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:08.362734    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:30:08.363062 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:08.363033    2561 patch_prober.go:28] interesting pod/e2e-ipp-a-068335-openshift-default-759456756d-rvspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Jul 31 22:30:08.363125 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:08.363092    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Jul 31 22:30:09.362848 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:09.362810    2561 patch_prober.go:28] interesting pod/e2e-ipp-a-068335-openshift-default-759456756d-rvspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Jul 31 22:30:09.363278 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:09.362872    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Jul 31 22:30:10.362278 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:10.362242    2561 patch_prober.go:28] interesting pod/e2e-ipp-a-068335-openshift-default-759456756d-rvspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Jul 31 22:30:10.362468 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:10.362301    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Jul 31 22:30:11.362314 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:11.362273    2561 patch_prober.go:28] interesting pod/e2e-ipp-a-068335-openshift-default-759456756d-rvspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Jul 31 22:30:11.362724 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:11.362333    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Jul 31 22:30:12.362582 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:12.362531    2561 patch_prober.go:28] interesting pod/e2e-ipp-a-068335-openshift-default-759456756d-rvspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Jul 31 22:30:12.363072 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:12.362610    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Jul 31 22:30:13.365400 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:13.365367    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:30:13.365811 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:13.365635    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:30:13.366460 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:13.366444    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:30:52.419865 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:30:52.419833    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-87xjd_6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4/payload-processing/0.log"
Jul 31 22:31:01.082479 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:01.082444    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:31:01.090295 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:01.090270    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:31:37.388382 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:37.388307    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-87xjd_6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4/payload-processing/0.log"
Jul 31 22:31:38.057978 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.057937    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"]
Jul 31 22:31:38.061871 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.061851    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.063924 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.063906    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-184ad7-openshift-default-dockercfg-9sbxw\""
Jul 31 22:31:38.074609 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.074582    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"]
Jul 31 22:31:38.163091 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.163053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-podinfo\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.163293 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.163110    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-credential-socket\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.163293 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.163223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrdb4\" (UniqueName: \"kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-kube-api-access-wrdb4\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.163293 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.163261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-data\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.163410 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.163354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-token\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.163410 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.163399    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-envoy\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.163475 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.163436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-certs\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.163475 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.163451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istiod-ca-cert\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.163543 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.163475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-socket\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.264276 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.264238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-envoy\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.264493 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.264315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-certs\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.264493 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.264344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istiod-ca-cert\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.264493 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.264378    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-socket\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.264493 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.264464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-podinfo\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.264732 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.264509    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-credential-socket\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.264732 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.264583    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrdb4\" (UniqueName: \"kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-kube-api-access-wrdb4\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.264732 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.264618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-data\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.264732 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.264668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-token\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.264948 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.264811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-certs\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.264948 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.264835    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-socket\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.265108 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.265072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-credential-socket\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.265430 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.265407    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-data\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.266129 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.266096    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istiod-ca-cert\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.267021 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.266987    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-envoy\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.267688 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.267663    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-podinfo\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.271957 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.271939    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-token\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.272190 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.272165    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrdb4\" (UniqueName: \"kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-kube-api-access-wrdb4\") pod \"e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") " pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.372632 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.372539    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:38.519362 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.519336    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"]
Jul 31 22:31:38.521626 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:31:38.521600    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe8adf2_4fde_4936_917f_7f0dcdd2b071.slice/crio-6998392d620a0b78dbb37d2d3fcb182126ddc6b6786a5104ca3fa75c2b3e3fd4 WatchSource:0}: Error finding container 6998392d620a0b78dbb37d2d3fcb182126ddc6b6786a5104ca3fa75c2b3e3fd4: Status 404 returned error can't find the container with id 6998392d620a0b78dbb37d2d3fcb182126ddc6b6786a5104ca3fa75c2b3e3fd4
Jul 31 22:31:38.524275 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.524238    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:31:38.524369 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.524305    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:31:38.524369 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.524332    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:31:38.530758 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:38.530735    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" event={"ID":"efe8adf2-4fde-4936-917f-7f0dcdd2b071","Type":"ContainerStarted","Data":"6998392d620a0b78dbb37d2d3fcb182126ddc6b6786a5104ca3fa75c2b3e3fd4"}
Jul 31 22:31:39.538170 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:39.538112    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" event={"ID":"efe8adf2-4fde-4936-917f-7f0dcdd2b071","Type":"ContainerStarted","Data":"157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c"}
Jul 31 22:31:39.556104 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:39.555988    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" podStartSLOduration=1.555968017 podStartE2EDuration="1.555968017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:31:39.555015945 +0000 UTC m=+3339.817764309" watchObservedRunningTime="2026-07-31 22:31:39.555968017 +0000 UTC m=+3339.818716382"
Jul 31 22:31:40.373569 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:40.373527    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:40.378953 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:40.378927    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:40.543397 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:40.543366    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:40.544387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:40.544365    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:56.585333 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:56.585284    2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 31 22:31:56.587617 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:56.585375    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 31 22:31:56.640167 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:56.640118    2561 generic.go:356] "Generic (PLEG): container finished" podID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerID="157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c" exitCode=0
Jul 31 22:31:56.640369 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:56.640163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" event={"ID":"efe8adf2-4fde-4936-917f-7f0dcdd2b071","Type":"ContainerDied","Data":"157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c"}
Jul 31 22:31:56.640665 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:56.640648    2561 scope.go:117] "RemoveContainer" containerID="157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c"
Jul 31 22:31:56.641844 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:56.641810    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:31:56.641933 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:56.641881    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:31:56.641976 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:56.641931    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 31 22:31:57.646700 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:57.646662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" event={"ID":"efe8adf2-4fde-4936-917f-7f0dcdd2b071","Type":"ContainerStarted","Data":"f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74"}
Jul 31 22:31:58.373446 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:58.373406    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:31:58.373639 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:58.373561    2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 31 22:31:58.373691 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:58.373629    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 31 22:31:59.373273 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:59.373238    2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 31 22:31:59.373648 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:31:59.373305    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 31 22:32:00.373944 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:00.373905    2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 31 22:32:00.374397 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:00.373971    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 31 22:32:01.373893 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:01.373855    2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 31 22:32:01.374090 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:01.373913    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 31 22:32:02.373566 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:02.373524    2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 31 22:32:02.373756 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:02.373584    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 31 22:32:03.376826 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:03.376794    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:32:03.377226 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:03.377076    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:32:03.377799 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:03.377780    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:32:29.872505 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:29.872469    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"]
Jul 31 22:32:29.873102 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:29.872778    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy" containerID="cri-o://f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74" gracePeriod=30
Jul 31 22:32:33.379376 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:33.379335    2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 22:32:33.379766 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:33.379415    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 22:32:35.131425 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.131389    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:32:35.256425 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256332    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-socket\") pod \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") "
Jul 31 22:32:35.256425 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256399    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-credential-socket\") pod \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") "
Jul 31 22:32:35.256425 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256419    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrdb4\" (UniqueName: \"kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-kube-api-access-wrdb4\") pod \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") "
Jul 31 22:32:35.256716 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256479    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-certs\") pod \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") "
Jul 31 22:32:35.256716 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256503    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-envoy\") pod \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") "
Jul 31 22:32:35.256716 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256682    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "efe8adf2-4fde-4936-917f-7f0dcdd2b071" (UID: "efe8adf2-4fde-4936-917f-7f0dcdd2b071"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:32:35.256875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256707    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-token\") pod \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") "
Jul 31 22:32:35.256875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256756    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "efe8adf2-4fde-4936-917f-7f0dcdd2b071" (UID: "efe8adf2-4fde-4936-917f-7f0dcdd2b071"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:32:35.256875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256801    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istiod-ca-cert\") pod \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") "
Jul 31 22:32:35.256875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256837    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-data\") pod \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") "
Jul 31 22:32:35.256875 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256873    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-podinfo\") pod \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\" (UID: \"efe8adf2-4fde-4936-917f-7f0dcdd2b071\") "
Jul 31 22:32:35.257216 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.256888    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "efe8adf2-4fde-4936-917f-7f0dcdd2b071" (UID: "efe8adf2-4fde-4936-917f-7f0dcdd2b071"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:32:35.257216 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.257094    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-data" (OuterVolumeSpecName: "istio-data") pod "efe8adf2-4fde-4936-917f-7f0dcdd2b071" (UID: "efe8adf2-4fde-4936-917f-7f0dcdd2b071"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:32:35.257216 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.257184    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "efe8adf2-4fde-4936-917f-7f0dcdd2b071" (UID: "efe8adf2-4fde-4936-917f-7f0dcdd2b071"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:32:35.257381 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.257322    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istiod-ca-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:32:35.257381 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.257341    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-data\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:32:35.257381 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.257354    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:32:35.257381 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.257366    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-credential-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:32:35.257381 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.257380    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-workload-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:32:35.258896 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.258866    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-kube-api-access-wrdb4" (OuterVolumeSpecName: "kube-api-access-wrdb4") pod "efe8adf2-4fde-4936-917f-7f0dcdd2b071" (UID: "efe8adf2-4fde-4936-917f-7f0dcdd2b071"). InnerVolumeSpecName "kube-api-access-wrdb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:32:35.259002 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.258934    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "efe8adf2-4fde-4936-917f-7f0dcdd2b071" (UID: "efe8adf2-4fde-4936-917f-7f0dcdd2b071"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 22:32:35.259207 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.259190    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "efe8adf2-4fde-4936-917f-7f0dcdd2b071" (UID: "efe8adf2-4fde-4936-917f-7f0dcdd2b071"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:32:35.259207 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.259197    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-token" (OuterVolumeSpecName: "istio-token") pod "efe8adf2-4fde-4936-917f-7f0dcdd2b071" (UID: "efe8adf2-4fde-4936-917f-7f0dcdd2b071"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:32:35.359414 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.359376    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrdb4\" (UniqueName: \"kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-kube-api-access-wrdb4\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:32:35.359414 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.359413    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-envoy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:32:35.359618 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.359428    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:32:35.359618 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.359442    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe8adf2-4fde-4936-917f-7f0dcdd2b071-istio-podinfo\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:32:35.846899 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.846864    2561 generic.go:356] "Generic (PLEG): container finished" podID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerID="f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74" exitCode=0
Jul 31 22:32:35.847097 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.846910    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"
Jul 31 22:32:35.847097 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.846947    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" event={"ID":"efe8adf2-4fde-4936-917f-7f0dcdd2b071","Type":"ContainerDied","Data":"f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74"}
Jul 31 22:32:35.847097 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.847003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8" event={"ID":"efe8adf2-4fde-4936-917f-7f0dcdd2b071","Type":"ContainerDied","Data":"6998392d620a0b78dbb37d2d3fcb182126ddc6b6786a5104ca3fa75c2b3e3fd4"}
Jul 31 22:32:35.847097 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.847026    2561 scope.go:117] "RemoveContainer" containerID="f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74"
Jul 31 22:32:35.856532 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.856516    2561 scope.go:117] "RemoveContainer" containerID="157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c"
Jul 31 22:32:35.865066 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.865049    2561 scope.go:117] "RemoveContainer" containerID="f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74"
Jul 31 22:32:35.865565 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:32:35.865419    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74\": container with ID starting with f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74 not found: ID does not exist" containerID="f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74"
Jul 31 22:32:35.865565 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.865480    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74"} err="failed to get container status \"f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74\": rpc error: code = NotFound desc = could not find container \"f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74\": container with ID starting with f952a623fa24297ec9f080f1ec1cf9f7a8eb617a4284e1b40c04af0b5dc76b74 not found: ID does not exist"
Jul 31 22:32:35.865565 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.865504    2561 scope.go:117] "RemoveContainer" containerID="157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c"
Jul 31 22:32:35.865995 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:32:35.865949    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c\": container with ID starting with 157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c not found: ID does not exist" containerID="157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c"
Jul 31 22:32:35.866076 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.866008    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c"} err="failed to get container status \"157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c\": rpc error: code = NotFound desc = could not find container \"157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c\": container with ID starting with 157d23aa2c860ca84adec2e1728ec176790074757facf5dfe5d8028129c0887c not found: ID does not exist"
Jul 31 22:32:35.868285 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.868265    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"]
Jul 31 22:32:35.870860 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:35.870843    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-184ad7-openshift-default-7d46695b66-dq2s8"]
Jul 31 22:32:36.229100 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:36.229001    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" path="/var/lib/kubelet/pods/efe8adf2-4fde-4936-917f-7f0dcdd2b071/volumes"
Jul 31 22:32:49.252716 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.252682    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"]
Jul 31 22:32:49.254054 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.253993    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r" podUID="221dd229-4d95-4c4e-996d-bc147db7af8b" containerName="payload-pre-processing" containerID="cri-o://ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601" gracePeriod=30
Jul 31 22:32:49.514936 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.514852    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:32:49.699843 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.699811    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221dd229-4d95-4c4e-996d-bc147db7af8b-plugins-config-volume\") pod \"221dd229-4d95-4c4e-996d-bc147db7af8b\" (UID: \"221dd229-4d95-4c4e-996d-bc147db7af8b\") "
Jul 31 22:32:49.700014 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.699958    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxfms\" (UniqueName: \"kubernetes.io/projected/221dd229-4d95-4c4e-996d-bc147db7af8b-kube-api-access-fxfms\") pod \"221dd229-4d95-4c4e-996d-bc147db7af8b\" (UID: \"221dd229-4d95-4c4e-996d-bc147db7af8b\") "
Jul 31 22:32:49.700288 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.700265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221dd229-4d95-4c4e-996d-bc147db7af8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "221dd229-4d95-4c4e-996d-bc147db7af8b" (UID: "221dd229-4d95-4c4e-996d-bc147db7af8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:32:49.700359 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.700345    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221dd229-4d95-4c4e-996d-bc147db7af8b-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:32:49.702017 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.701995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221dd229-4d95-4c4e-996d-bc147db7af8b-kube-api-access-fxfms" (OuterVolumeSpecName: "kube-api-access-fxfms") pod "221dd229-4d95-4c4e-996d-bc147db7af8b" (UID: "221dd229-4d95-4c4e-996d-bc147db7af8b"). InnerVolumeSpecName "kube-api-access-fxfms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:32:49.801835 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.801799    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxfms\" (UniqueName: \"kubernetes.io/projected/221dd229-4d95-4c4e-996d-bc147db7af8b-kube-api-access-fxfms\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:32:49.923872 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.923836    2561 generic.go:356] "Generic (PLEG): container finished" podID="221dd229-4d95-4c4e-996d-bc147db7af8b" containerID="ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601" exitCode=0
Jul 31 22:32:49.924082 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.923883    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"
Jul 31 22:32:49.924082 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.923924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r" event={"ID":"221dd229-4d95-4c4e-996d-bc147db7af8b","Type":"ContainerDied","Data":"ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601"}
Jul 31 22:32:49.924082 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.923991    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r" event={"ID":"221dd229-4d95-4c4e-996d-bc147db7af8b","Type":"ContainerDied","Data":"3e01cda2065de24b9e4946eee2232e948a52fc30c73b2f22121807f27a386d53"}
Jul 31 22:32:49.924082 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.924023    2561 scope.go:117] "RemoveContainer" containerID="ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601"
Jul 31 22:32:49.932821 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.932801    2561 scope.go:117] "RemoveContainer" containerID="ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601"
Jul 31 22:32:49.933103 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:32:49.933084    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601\": container with ID starting with ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601 not found: ID does not exist" containerID="ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601"
Jul 31 22:32:49.933184 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.933113    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601"} err="failed to get container status \"ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601\": rpc error: code = NotFound desc = could not find container \"ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601\": container with ID starting with ac7459568f2fae775e59c6c8fb6d680ffb39d45e1829ea40d72eb9d67935e601 not found: ID does not exist"
Jul 31 22:32:49.944828 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.944800    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"]
Jul 31 22:32:49.947067 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:49.947043    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-068335-6b6b87ff6-sn28r"]
Jul 31 22:32:50.230397 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:32:50.230309    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="221dd229-4d95-4c4e-996d-bc147db7af8b" path="/var/lib/kubelet/pods/221dd229-4d95-4c4e-996d-bc147db7af8b/volumes"
Jul 31 22:33:52.273568 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:52.273534    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"]
Jul 31 22:33:52.274167 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:52.273808    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerName="istio-proxy" containerID="cri-o://5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8" gracePeriod=30
Jul 31 22:33:57.533446 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.533417    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:33:57.677796 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.677702    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-certs\") pod \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") "
Jul 31 22:33:57.677963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.677813    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-envoy\") pod \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") "
Jul 31 22:33:57.677963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.677840    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-socket\") pod \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") "
Jul 31 22:33:57.677963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.677885    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istiod-ca-cert\") pod \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") "
Jul 31 22:33:57.677963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.677923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-podinfo\") pod \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") "
Jul 31 22:33:57.677963 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.677951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-credential-socket\") pod \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") "
Jul 31 22:33:57.678271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678026    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" (UID: "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:33:57.678271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678093    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngbt2\" (UniqueName: \"kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-kube-api-access-ngbt2\") pod \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") "
Jul 31 22:33:57.678271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678180    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-token\") pod \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") "
Jul 31 22:33:57.678271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678186    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" (UID: "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:33:57.678271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678216    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" (UID: "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:33:57.678271 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678237    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-data\") pod \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\" (UID: \"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad\") "
Jul 31 22:33:57.678553 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678352    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" (UID: "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:33:57.678553 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678481    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" (UID: "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:33:57.678633 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678611    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:33:57.678685 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678630    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istiod-ca-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:33:57.678685 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678644    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-credential-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:33:57.678685 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678657    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-data\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:33:57.678685 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.678670    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-workload-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:33:57.680350 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.680324    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" (UID: "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:33:57.680350 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.680342    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-kube-api-access-ngbt2" (OuterVolumeSpecName: "kube-api-access-ngbt2") pod "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" (UID: "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad"). InnerVolumeSpecName "kube-api-access-ngbt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:33:57.680569 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.680545    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" (UID: "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 22:33:57.680642 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.680610    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" (UID: "527acc4e-c0bb-4f02-9bf0-3bd71e5598ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:33:57.779237 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.779200    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-envoy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:33:57.779237 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.779234    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-podinfo\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:33:57.779237 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.779245    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngbt2\" (UniqueName: \"kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-kube-api-access-ngbt2\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:33:57.779498 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:57.779259    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad-istio-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:33:58.273549 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.273515    2561 generic.go:356] "Generic (PLEG): container finished" podID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerID="5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8" exitCode=0
Jul 31 22:33:58.273750 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.273569    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"
Jul 31 22:33:58.273750 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.273597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" event={"ID":"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad","Type":"ContainerDied","Data":"5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8"}
Jul 31 22:33:58.273750 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.273650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz" event={"ID":"527acc4e-c0bb-4f02-9bf0-3bd71e5598ad","Type":"ContainerDied","Data":"b5348ce2f04be4a8734c2d025ebc48ac940245376c6a66697f32823a1f0755dd"}
Jul 31 22:33:58.273750 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.273673    2561 scope.go:117] "RemoveContainer" containerID="5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8"
Jul 31 22:33:58.284674 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.284620    2561 scope.go:117] "RemoveContainer" containerID="46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048"
Jul 31 22:33:58.289683 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.289650    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"]
Jul 31 22:33:58.292384 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.292362    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-068335-openshift-default-759456756d-rvspz"]
Jul 31 22:33:58.293394 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.293377    2561 scope.go:117] "RemoveContainer" containerID="5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8"
Jul 31 22:33:58.293648 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:33:58.293628    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8\": container with ID starting with 5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8 not found: ID does not exist" containerID="5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8"
Jul 31 22:33:58.293699 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.293656    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8"} err="failed to get container status \"5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8\": rpc error: code = NotFound desc = could not find container \"5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8\": container with ID starting with 5b2eed6604eee5edd25ae8d97b98d57458b3c3cf6681a3ba6b619c24e4bb32b8 not found: ID does not exist"
Jul 31 22:33:58.293699 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.293675    2561 scope.go:117] "RemoveContainer" containerID="46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048"
Jul 31 22:33:58.293923 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:33:58.293906    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048\": container with ID starting with 46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048 not found: ID does not exist" containerID="46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048"
Jul 31 22:33:58.293968 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:33:58.293929    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048"} err="failed to get container status \"46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048\": rpc error: code = NotFound desc = could not find container \"46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048\": container with ID starting with 46ca4acd3fc3b42048d8e6f43e275b1a88eb56d90bdc055848f220d9944c1048 not found: ID does not exist"
Jul 31 22:34:00.228185 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:00.228151    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" path="/var/lib/kubelet/pods/527acc4e-c0bb-4f02-9bf0-3bd71e5598ad/volumes"
Jul 31 22:34:20.551330 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:20.551291    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"]
Jul 31 22:34:20.551883 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:20.551572    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9" podUID="f7e2b429-211b-48b7-9460-0fd73202bef3" containerName="maas-api" containerID="cri-o://5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9" gracePeriod=30
Jul 31 22:34:20.615393 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:20.615352    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"]
Jul 31 22:34:20.615626 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:20.615595    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc" podUID="c4c3291d-abfd-4f5c-a402-eea21758eafc" containerName="payload-pre-processing" containerID="cri-o://111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a" gracePeriod=30
Jul 31 22:34:20.872556 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:20.872522    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:34:20.905540 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:20.905512    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:34:20.999685 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:20.999644    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e2b429-211b-48b7-9460-0fd73202bef3-openshift-service-ca\") pod \"f7e2b429-211b-48b7-9460-0fd73202bef3\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") "
Jul 31 22:34:20.999988 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:20.999715    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npmhm\" (UniqueName: \"kubernetes.io/projected/f7e2b429-211b-48b7-9460-0fd73202bef3-kube-api-access-npmhm\") pod \"f7e2b429-211b-48b7-9460-0fd73202bef3\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") "
Jul 31 22:34:20.999988 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:20.999743    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87m8d\" (UniqueName: \"kubernetes.io/projected/c4c3291d-abfd-4f5c-a402-eea21758eafc-kube-api-access-87m8d\") pod \"c4c3291d-abfd-4f5c-a402-eea21758eafc\" (UID: \"c4c3291d-abfd-4f5c-a402-eea21758eafc\") "
Jul 31 22:34:20.999988 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:20.999790    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c3291d-abfd-4f5c-a402-eea21758eafc-plugins-config-volume\") pod \"c4c3291d-abfd-4f5c-a402-eea21758eafc\" (UID: \"c4c3291d-abfd-4f5c-a402-eea21758eafc\") "
Jul 31 22:34:20.999988 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:20.999873    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e2b429-211b-48b7-9460-0fd73202bef3-maas-api-tls\") pod \"f7e2b429-211b-48b7-9460-0fd73202bef3\" (UID: \"f7e2b429-211b-48b7-9460-0fd73202bef3\") "
Jul 31 22:34:21.000243 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.000087    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e2b429-211b-48b7-9460-0fd73202bef3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7e2b429-211b-48b7-9460-0fd73202bef3" (UID: "f7e2b429-211b-48b7-9460-0fd73202bef3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:34:21.000286 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.000238    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c3291d-abfd-4f5c-a402-eea21758eafc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4c3291d-abfd-4f5c-a402-eea21758eafc" (UID: "c4c3291d-abfd-4f5c-a402-eea21758eafc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:34:21.000330 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.000303    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e2b429-211b-48b7-9460-0fd73202bef3-openshift-service-ca\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:21.001959 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.001935    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e2b429-211b-48b7-9460-0fd73202bef3-kube-api-access-npmhm" (OuterVolumeSpecName: "kube-api-access-npmhm") pod "f7e2b429-211b-48b7-9460-0fd73202bef3" (UID: "f7e2b429-211b-48b7-9460-0fd73202bef3"). InnerVolumeSpecName "kube-api-access-npmhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:34:21.002053 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.001934    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e2b429-211b-48b7-9460-0fd73202bef3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7e2b429-211b-48b7-9460-0fd73202bef3" (UID: "f7e2b429-211b-48b7-9460-0fd73202bef3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 22:34:21.002053 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.001959    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c3291d-abfd-4f5c-a402-eea21758eafc-kube-api-access-87m8d" (OuterVolumeSpecName: "kube-api-access-87m8d") pod "c4c3291d-abfd-4f5c-a402-eea21758eafc" (UID: "c4c3291d-abfd-4f5c-a402-eea21758eafc"). InnerVolumeSpecName "kube-api-access-87m8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:34:21.101488 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.101396    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e2b429-211b-48b7-9460-0fd73202bef3-maas-api-tls\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:21.101488 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.101431    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npmhm\" (UniqueName: \"kubernetes.io/projected/f7e2b429-211b-48b7-9460-0fd73202bef3-kube-api-access-npmhm\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:21.101488 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.101442    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87m8d\" (UniqueName: \"kubernetes.io/projected/c4c3291d-abfd-4f5c-a402-eea21758eafc-kube-api-access-87m8d\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:21.101488 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.101451    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c3291d-abfd-4f5c-a402-eea21758eafc-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:21.390077 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.389987    2561 generic.go:356] "Generic (PLEG): container finished" podID="c4c3291d-abfd-4f5c-a402-eea21758eafc" containerID="111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a" exitCode=0
Jul 31 22:34:21.390077 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.390034    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"
Jul 31 22:34:21.390077 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.390056    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc" event={"ID":"c4c3291d-abfd-4f5c-a402-eea21758eafc","Type":"ContainerDied","Data":"111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a"}
Jul 31 22:34:21.390368 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.390111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc" event={"ID":"c4c3291d-abfd-4f5c-a402-eea21758eafc","Type":"ContainerDied","Data":"c41717ac07c77e8df9d5e4112bd8edd95176a73258f9f5a4dff0ad64f400cd1f"}
Jul 31 22:34:21.390368 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.390178    2561 scope.go:117] "RemoveContainer" containerID="111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a"
Jul 31 22:34:21.391607 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.391478    2561 generic.go:356] "Generic (PLEG): container finished" podID="f7e2b429-211b-48b7-9460-0fd73202bef3" containerID="5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9" exitCode=0
Jul 31 22:34:21.391607 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.391506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9" event={"ID":"f7e2b429-211b-48b7-9460-0fd73202bef3","Type":"ContainerDied","Data":"5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9"}
Jul 31 22:34:21.391607 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.391534    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"
Jul 31 22:34:21.391607 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.391562    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9" event={"ID":"f7e2b429-211b-48b7-9460-0fd73202bef3","Type":"ContainerDied","Data":"0c8c474b21335a49efa1b09d3e18e76d3930c1659add247bec4689b85ca8b891"}
Jul 31 22:34:21.400090 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.400071    2561 scope.go:117] "RemoveContainer" containerID="111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a"
Jul 31 22:34:21.400418 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:34:21.400396    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a\": container with ID starting with 111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a not found: ID does not exist" containerID="111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a"
Jul 31 22:34:21.400478 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.400426    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a"} err="failed to get container status \"111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a\": rpc error: code = NotFound desc = could not find container \"111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a\": container with ID starting with 111725dde301a2837ba457e3d1251584792d3986f59846fde55df5c1c37ccc6a not found: ID does not exist"
Jul 31 22:34:21.400478 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.400443    2561 scope.go:117] "RemoveContainer" containerID="5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9"
Jul 31 22:34:21.408372 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.408353    2561 scope.go:117] "RemoveContainer" containerID="5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9"
Jul 31 22:34:21.408641 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:34:21.408623    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9\": container with ID starting with 5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9 not found: ID does not exist" containerID="5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9"
Jul 31 22:34:21.408693 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.408648    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9"} err="failed to get container status \"5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9\": rpc error: code = NotFound desc = could not find container \"5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9\": container with ID starting with 5367c607f7aec39910ed4681c90ac3625a5ef05c7c0a4f340f8e6a89113790d9 not found: ID does not exist"
Jul 31 22:34:21.412576 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.412555    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"]
Jul 31 22:34:21.416791 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.416765    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ee9929-8475679db6-pkvf9"]
Jul 31 22:34:21.426316 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.426287    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"]
Jul 31 22:34:21.429833 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:21.429802    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ee9929-588d49d4cb-kcrcc"]
Jul 31 22:34:22.227662 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:22.227628    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4c3291d-abfd-4f5c-a402-eea21758eafc" path="/var/lib/kubelet/pods/c4c3291d-abfd-4f5c-a402-eea21758eafc/volumes"
Jul 31 22:34:22.228053 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:22.228028    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e2b429-211b-48b7-9460-0fd73202bef3" path="/var/lib/kubelet/pods/f7e2b429-211b-48b7-9460-0fd73202bef3/volumes"
Jul 31 22:34:27.808077 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:27.808042    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"]
Jul 31 22:34:27.808601 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:27.808272    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy" containerID="cri-o://b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0" gracePeriod=30
Jul 31 22:34:30.273310 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:30.273268    2561 patch_prober.go:28] interesting pod/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 22:34:30.273702 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:30.273331    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 22:34:33.080684 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.080611    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:34:33.217450 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.217417    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-envoy\") pod \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") "
Jul 31 22:34:33.217657 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.217525    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-data\") pod \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") "
Jul 31 22:34:33.217657 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.217557    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-token\") pod \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") "
Jul 31 22:34:33.217657 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.217586    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-podinfo\") pod \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") "
Jul 31 22:34:33.217657 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.217632    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-credential-socket\") pod \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") "
Jul 31 22:34:33.217884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.217725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-certs\") pod \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") "
Jul 31 22:34:33.217884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.217760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-socket\") pod \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") "
Jul 31 22:34:33.217884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.217788    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9j9f\" (UniqueName: \"kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-kube-api-access-q9j9f\") pod \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") "
Jul 31 22:34:33.217884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.217815    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istiod-ca-cert\") pod \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\" (UID: \"31b357f0-1a81-4d91-9ed2-10db5b9f4a29\") "
Jul 31 22:34:33.217884 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.217845    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-data" (OuterVolumeSpecName: "istio-data") pod "31b357f0-1a81-4d91-9ed2-10db5b9f4a29" (UID: "31b357f0-1a81-4d91-9ed2-10db5b9f4a29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:34:33.218171 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.217974    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "31b357f0-1a81-4d91-9ed2-10db5b9f4a29" (UID: "31b357f0-1a81-4d91-9ed2-10db5b9f4a29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:34:33.218171 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.218051    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "31b357f0-1a81-4d91-9ed2-10db5b9f4a29" (UID: "31b357f0-1a81-4d91-9ed2-10db5b9f4a29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:34:33.218299 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.218239    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-data\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:33.218299 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.218262    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-credential-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:33.218299 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.218279    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:33.218459 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.218357    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "31b357f0-1a81-4d91-9ed2-10db5b9f4a29" (UID: "31b357f0-1a81-4d91-9ed2-10db5b9f4a29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:34:33.218459 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.218390    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "31b357f0-1a81-4d91-9ed2-10db5b9f4a29" (UID: "31b357f0-1a81-4d91-9ed2-10db5b9f4a29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:34:33.219830 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.219802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "31b357f0-1a81-4d91-9ed2-10db5b9f4a29" (UID: "31b357f0-1a81-4d91-9ed2-10db5b9f4a29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:34:33.219958 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.219860    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "31b357f0-1a81-4d91-9ed2-10db5b9f4a29" (UID: "31b357f0-1a81-4d91-9ed2-10db5b9f4a29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 22:34:33.220198 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.220176    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-token" (OuterVolumeSpecName: "istio-token") pod "31b357f0-1a81-4d91-9ed2-10db5b9f4a29" (UID: "31b357f0-1a81-4d91-9ed2-10db5b9f4a29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:34:33.220339 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.220310    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-kube-api-access-q9j9f" (OuterVolumeSpecName: "kube-api-access-q9j9f") pod "31b357f0-1a81-4d91-9ed2-10db5b9f4a29" (UID: "31b357f0-1a81-4d91-9ed2-10db5b9f4a29"). InnerVolumeSpecName "kube-api-access-q9j9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:34:33.319376 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.319334    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-workload-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:33.319376 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.319369    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9j9f\" (UniqueName: \"kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-kube-api-access-q9j9f\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:33.319376 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.319380    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istiod-ca-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:33.319619 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.319390    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-envoy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:33.319619 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.319398    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:33.319619 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.319408    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b357f0-1a81-4d91-9ed2-10db5b9f4a29-istio-podinfo\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:33.457573 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.457475    2561 generic.go:356] "Generic (PLEG): container finished" podID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerID="b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0" exitCode=0
Jul 31 22:34:33.457573 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.457527    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"
Jul 31 22:34:33.457573 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.457556    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" event={"ID":"31b357f0-1a81-4d91-9ed2-10db5b9f4a29","Type":"ContainerDied","Data":"b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0"}
Jul 31 22:34:33.457826 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.457611    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7" event={"ID":"31b357f0-1a81-4d91-9ed2-10db5b9f4a29","Type":"ContainerDied","Data":"462bd79286a49431e4b25d30ced0ac5c163f8816c269b8495ef54870961c4836"}
Jul 31 22:34:33.457826 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.457635    2561 scope.go:117] "RemoveContainer" containerID="b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0"
Jul 31 22:34:33.467349 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.467321    2561 scope.go:117] "RemoveContainer" containerID="026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4"
Jul 31 22:34:33.475933 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.475911    2561 scope.go:117] "RemoveContainer" containerID="b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0"
Jul 31 22:34:33.476275 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:34:33.476255    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0\": container with ID starting with b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0 not found: ID does not exist" containerID="b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0"
Jul 31 22:34:33.476349 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.476286    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0"} err="failed to get container status \"b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0\": rpc error: code = NotFound desc = could not find container \"b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0\": container with ID starting with b18b26fa79ddba4ed3d9db89de71659ecc3e00a8dd8cd1a632b44d49e4370bb0 not found: ID does not exist"
Jul 31 22:34:33.476349 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.476306    2561 scope.go:117] "RemoveContainer" containerID="026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4"
Jul 31 22:34:33.476570 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:34:33.476544    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4\": container with ID starting with 026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4 not found: ID does not exist" containerID="026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4"
Jul 31 22:34:33.476663 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.476580    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4"} err="failed to get container status \"026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4\": rpc error: code = NotFound desc = could not find container \"026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4\": container with ID starting with 026967773e2bf55e89fe8a7c2aef051609c620fc1d36dd4bab5dea9bcbea1bf4 not found: ID does not exist"
Jul 31 22:34:33.479325 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.479298    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"]
Jul 31 22:34:33.482593 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:33.482569    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ee9929-openshift-default-6c77cddd48-xqqj7"]
Jul 31 22:34:34.227534 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:34.227496    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" path="/var/lib/kubelet/pods/31b357f0-1a81-4d91-9ed2-10db5b9f4a29/volumes"
Jul 31 22:34:38.891525 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:38.889679    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"]
Jul 31 22:34:38.891525 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:38.889966    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p" podUID="af975e5f-aa5e-4237-9818-1c8a6f134748" containerName="payload-pre-processing" containerID="cri-o://3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d" gracePeriod=30
Jul 31 22:34:39.145842 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.145771    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:34:39.277099 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.277058    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af975e5f-aa5e-4237-9818-1c8a6f134748-plugins-config-volume\") pod \"af975e5f-aa5e-4237-9818-1c8a6f134748\" (UID: \"af975e5f-aa5e-4237-9818-1c8a6f134748\") "
Jul 31 22:34:39.277321 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.277122    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlsc7\" (UniqueName: \"kubernetes.io/projected/af975e5f-aa5e-4237-9818-1c8a6f134748-kube-api-access-xlsc7\") pod \"af975e5f-aa5e-4237-9818-1c8a6f134748\" (UID: \"af975e5f-aa5e-4237-9818-1c8a6f134748\") "
Jul 31 22:34:39.277550 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.277521    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af975e5f-aa5e-4237-9818-1c8a6f134748-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af975e5f-aa5e-4237-9818-1c8a6f134748" (UID: "af975e5f-aa5e-4237-9818-1c8a6f134748"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:34:39.279349 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.279316    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af975e5f-aa5e-4237-9818-1c8a6f134748-kube-api-access-xlsc7" (OuterVolumeSpecName: "kube-api-access-xlsc7") pod "af975e5f-aa5e-4237-9818-1c8a6f134748" (UID: "af975e5f-aa5e-4237-9818-1c8a6f134748"). InnerVolumeSpecName "kube-api-access-xlsc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:34:39.378407 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.378361    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af975e5f-aa5e-4237-9818-1c8a6f134748-plugins-config-volume\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:39.378407 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.378396    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlsc7\" (UniqueName: \"kubernetes.io/projected/af975e5f-aa5e-4237-9818-1c8a6f134748-kube-api-access-xlsc7\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:39.487867 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.487776    2561 generic.go:356] "Generic (PLEG): container finished" podID="af975e5f-aa5e-4237-9818-1c8a6f134748" containerID="3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d" exitCode=0
Jul 31 22:34:39.487867 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.487821    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"
Jul 31 22:34:39.488060 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.487858    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p" event={"ID":"af975e5f-aa5e-4237-9818-1c8a6f134748","Type":"ContainerDied","Data":"3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d"}
Jul 31 22:34:39.488060 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.487916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p" event={"ID":"af975e5f-aa5e-4237-9818-1c8a6f134748","Type":"ContainerDied","Data":"57d8d25c362991826946dd33e693450e59d42a6a1539934ab58025cd4f6c5bba"}
Jul 31 22:34:39.488060 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.487939    2561 scope.go:117] "RemoveContainer" containerID="3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d"
Jul 31 22:34:39.497054 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.497037    2561 scope.go:117] "RemoveContainer" containerID="3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d"
Jul 31 22:34:39.497361 ip-10-0-135-241 kubenswrapper[2561]: E0731 22:34:39.497341    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d\": container with ID starting with 3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d not found: ID does not exist" containerID="3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d"
Jul 31 22:34:39.497435 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.497369    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d"} err="failed to get container status \"3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d\": rpc error: code = NotFound desc = could not find container \"3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d\": container with ID starting with 3a60aa4517f4a71c5220ceedabceec44d41006e378ca048f1aac5d786de07b7d not found: ID does not exist"
Jul 31 22:34:39.508316 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.508286    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"]
Jul 31 22:34:39.518438 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:39.518411    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-21e950-56c86dcbbd-sjr5p"]
Jul 31 22:34:40.228727 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:40.228694    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af975e5f-aa5e-4237-9818-1c8a6f134748" path="/var/lib/kubelet/pods/af975e5f-aa5e-4237-9818-1c8a6f134748/volumes"
Jul 31 22:34:45.407668 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:45.407628    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"]
Jul 31 22:34:45.408512 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:45.408445    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerName="istio-proxy" containerID="cri-o://4bf2419d7407498b356def4006f6b81d2f2446bcedc06838224b0ac7dece8aa0" gracePeriod=30
Jul 31 22:34:46.121417 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:46.121380    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-678d57fdb7-59l5f_1d36e5c8-8d22-4825-8bdb-a04df6c948f5/authorino/0.log"
Jul 31 22:34:50.543889 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.543859    2561 generic.go:356] "Generic (PLEG): container finished" podID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerID="4bf2419d7407498b356def4006f6b81d2f2446bcedc06838224b0ac7dece8aa0" exitCode=0
Jul 31 22:34:50.544303 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.543945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" event={"ID":"d857d9e1-caf3-42a1-b68e-5a6e9427e23b","Type":"ContainerDied","Data":"4bf2419d7407498b356def4006f6b81d2f2446bcedc06838224b0ac7dece8aa0"}
Jul 31 22:34:50.544303 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.543998    2561 scope.go:117] "RemoveContainer" containerID="d36e883372f6f1671fc0150ff66a34fe33b713e373548d2234d8880e9b875366"
Jul 31 22:34:50.663856 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.663830    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:34:50.680078 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680043    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dh5t\" (UniqueName: \"kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-kube-api-access-6dh5t\") pod \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") "
Jul 31 22:34:50.680268 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680098    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-socket\") pod \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") "
Jul 31 22:34:50.680268 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680192    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-podinfo\") pod \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") "
Jul 31 22:34:50.680268 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680246    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-certs\") pod \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") "
Jul 31 22:34:50.680435 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680300    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istiod-ca-cert\") pod \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") "
Jul 31 22:34:50.680435 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680328    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-data\") pod \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") "
Jul 31 22:34:50.680435 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680385    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-token\") pod \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") "
Jul 31 22:34:50.680589 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680441    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-credential-socket\") pod \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") "
Jul 31 22:34:50.680589 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680483    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-envoy\") pod \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\" (UID: \"d857d9e1-caf3-42a1-b68e-5a6e9427e23b\") "
Jul 31 22:34:50.680589 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680487    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d857d9e1-caf3-42a1-b68e-5a6e9427e23b" (UID: "d857d9e1-caf3-42a1-b68e-5a6e9427e23b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:34:50.680888 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680699    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-data" (OuterVolumeSpecName: "istio-data") pod "d857d9e1-caf3-42a1-b68e-5a6e9427e23b" (UID: "d857d9e1-caf3-42a1-b68e-5a6e9427e23b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:34:50.681398 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.680807    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:50.681618 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.681553    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d857d9e1-caf3-42a1-b68e-5a6e9427e23b" (UID: "d857d9e1-caf3-42a1-b68e-5a6e9427e23b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:34:50.681618 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.681600    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d857d9e1-caf3-42a1-b68e-5a6e9427e23b" (UID: "d857d9e1-caf3-42a1-b68e-5a6e9427e23b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 22:34:50.682698 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.682660    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d857d9e1-caf3-42a1-b68e-5a6e9427e23b" (UID: "d857d9e1-caf3-42a1-b68e-5a6e9427e23b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:34:50.683262 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.683230    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-kube-api-access-6dh5t" (OuterVolumeSpecName: "kube-api-access-6dh5t") pod "d857d9e1-caf3-42a1-b68e-5a6e9427e23b" (UID: "d857d9e1-caf3-42a1-b68e-5a6e9427e23b"). InnerVolumeSpecName "kube-api-access-6dh5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:34:50.685043 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.684459    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d857d9e1-caf3-42a1-b68e-5a6e9427e23b" (UID: "d857d9e1-caf3-42a1-b68e-5a6e9427e23b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 22:34:50.685691 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.685671    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d857d9e1-caf3-42a1-b68e-5a6e9427e23b" (UID: "d857d9e1-caf3-42a1-b68e-5a6e9427e23b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 22:34:50.686086 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.686063    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-token" (OuterVolumeSpecName: "istio-token") pod "d857d9e1-caf3-42a1-b68e-5a6e9427e23b" (UID: "d857d9e1-caf3-42a1-b68e-5a6e9427e23b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 22:34:50.781425 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.781379    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-envoy\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:50.781425 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.781416    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dh5t\" (UniqueName: \"kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-kube-api-access-6dh5t\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:50.781425 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.781426    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-podinfo\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:50.781425 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.781435    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-workload-certs\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:50.781691 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.781444    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istiod-ca-cert\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:50.781691 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.781451    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-data\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:50.781691 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.781458    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-istio-token\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:50.781691 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:50.781466    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d857d9e1-caf3-42a1-b68e-5a6e9427e23b-credential-socket\") on node \"ip-10-0-135-241.ec2.internal\" DevicePath \"\""
Jul 31 22:34:51.550338 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:51.550306    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"
Jul 31 22:34:51.550824 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:51.550345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8" event={"ID":"d857d9e1-caf3-42a1-b68e-5a6e9427e23b","Type":"ContainerDied","Data":"0cc134f3f3921014b6f81a366dfa48acc7cefce53e719e8dbfe4c3e72afac179"}
Jul 31 22:34:51.550824 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:51.550398    2561 scope.go:117] "RemoveContainer" containerID="4bf2419d7407498b356def4006f6b81d2f2446bcedc06838224b0ac7dece8aa0"
Jul 31 22:34:51.576968 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:51.576935    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"]
Jul 31 22:34:51.580071 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:51.580044    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-21e950-openshift-default-d8d4fd85b-fhwb8"]
Jul 31 22:34:51.648412 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:51.648375    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79f8b57f55-dm5zm_a98c3c97-149b-4886-aa32-8708d6f1a775/manager/0.log"
Jul 31 22:34:52.017991 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:52.017905    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67685b9c4d-gnpfk_e837a855-7075-4797-8c7e-a8ac8f2ab514/copy-manifests/0.log"
Jul 31 22:34:52.036057 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:52.036025    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67685b9c4d-gnpfk_e837a855-7075-4797-8c7e-a8ac8f2ab514/manager/0.log"
Jul 31 22:34:52.227800 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:52.227769    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" path="/var/lib/kubelet/pods/d857d9e1-caf3-42a1-b68e-5a6e9427e23b/volumes"
Jul 31 22:34:52.681025 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:52.680994    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7647b7b95c-2tbzg_a215b518-9422-4eae-8aad-5dcd3fa1d612/maas-api/0.log"
Jul 31 22:34:53.929696 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:53.929661    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-678d57fdb7-59l5f_1d36e5c8-8d22-4825-8bdb-a04df6c948f5/authorino/0.log"
Jul 31 22:34:55.224727 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:55.224698    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64667dffd7-bl6qh_3963e755-e179-43ca-9d44-0a1c1dd6e0bd/kube-auth-proxy/0.log"
Jul 31 22:34:55.584129 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:55.584095    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-z4jdf_2b12a163-fd17-4b59-a3a4-c9397968d10a/payload-pre-processing/0.log"
Jul 31 22:34:55.708627 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:55.708595    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-87xjd_6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4/payload-processing/0.log"
Jul 31 22:34:56.184164 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:56.184114    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-zmcjn_3fa59a92-b501-4886-9739-04df669aeb1e/main/0.log"
Jul 31 22:34:56.301223 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:34:56.301192    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-8b87k_bbf98ded-6e86-4202-9b13-d1b9153e3a22/main/0.log"
Jul 31 22:35:05.809400 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:05.809366    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rw7xc_2d760f73-be89-4905-88ed-68554584ed68/global-pull-secret-syncer/0.log"
Jul 31 22:35:05.974825 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:05.974790    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nfmj9_3166d1ee-0352-46a8-a585-0189cba32bd6/konnectivity-agent/0.log"
Jul 31 22:35:06.020313 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:06.020276    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-241.ec2.internal_b97b9740959824207498b954ade89901/haproxy/0.log"
Jul 31 22:35:11.354785 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:11.354736    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-678d57fdb7-59l5f_1d36e5c8-8d22-4825-8bdb-a04df6c948f5/authorino/0.log"
Jul 31 22:35:13.390706 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.390622    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-8p8cq_ce19efed-c49f-4620-abc9-6f312195bff0/cluster-monitoring-operator/0.log"
Jul 31 22:35:13.493483 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.493440    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-86b995f864-xcxh9_648ef71b-d43f-4064-8809-eab70495c995/metrics-server/0.log"
Jul 31 22:35:13.515589 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.515558    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-t6c86_59e2e80a-f0c0-4a29-9238-13ecfff3e8c5/monitoring-plugin/0.log"
Jul 31 22:35:13.558370 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.558339    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nhjr_1845b24a-5171-4694-9185-4cb29e3b1298/node-exporter/0.log"
Jul 31 22:35:13.578177 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.578152    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nhjr_1845b24a-5171-4694-9185-4cb29e3b1298/kube-rbac-proxy/0.log"
Jul 31 22:35:13.597045 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.597022    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nhjr_1845b24a-5171-4694-9185-4cb29e3b1298/init-textfile/0.log"
Jul 31 22:35:13.759091 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.759006    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wtx6h_a92fc535-e796-42d3-a398-84343d498630/kube-rbac-proxy-main/0.log"
Jul 31 22:35:13.777504 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.777475    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wtx6h_a92fc535-e796-42d3-a398-84343d498630/kube-rbac-proxy-self/0.log"
Jul 31 22:35:13.796078 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.796053    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wtx6h_a92fc535-e796-42d3-a398-84343d498630/openshift-state-metrics/0.log"
Jul 31 22:35:13.829817 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.829790    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d88fee0-7666-4359-9724-4724c449c3bd/prometheus/0.log"
Jul 31 22:35:13.853930 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.853904    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d88fee0-7666-4359-9724-4724c449c3bd/config-reloader/0.log"
Jul 31 22:35:13.876199 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.876171    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d88fee0-7666-4359-9724-4724c449c3bd/thanos-sidecar/0.log"
Jul 31 22:35:13.894308 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.894288    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d88fee0-7666-4359-9724-4724c449c3bd/kube-rbac-proxy-web/0.log"
Jul 31 22:35:13.912755 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.912726    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d88fee0-7666-4359-9724-4724c449c3bd/kube-rbac-proxy/0.log"
Jul 31 22:35:13.932627 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.932600    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d88fee0-7666-4359-9724-4724c449c3bd/kube-rbac-proxy-thanos/0.log"
Jul 31 22:35:13.957146 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:13.957110    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d88fee0-7666-4359-9724-4724c449c3bd/init-config-reloader/0.log"
Jul 31 22:35:15.453928 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.453891    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"]
Jul 31 22:35:15.454297 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454258    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerName="istio-proxy"
Jul 31 22:35:15.454297 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454271    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerName="istio-proxy"
Jul 31 22:35:15.454297 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454278    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy"
Jul 31 22:35:15.454297 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454284    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy"
Jul 31 22:35:15.454297 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454298    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e2b429-211b-48b7-9460-0fd73202bef3" containerName="maas-api"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454304    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e2b429-211b-48b7-9460-0fd73202bef3" containerName="maas-api"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454314    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerName="istio-proxy"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454320    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerName="istio-proxy"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454328    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4c3291d-abfd-4f5c-a402-eea21758eafc" containerName="payload-pre-processing"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454333    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4c3291d-abfd-4f5c-a402-eea21758eafc" containerName="payload-pre-processing"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454340    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454348    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454362    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454371    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454383    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af975e5f-aa5e-4237-9818-1c8a6f134748" containerName="payload-pre-processing"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454389    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="af975e5f-aa5e-4237-9818-1c8a6f134748" containerName="payload-pre-processing"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454396    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221dd229-4d95-4c4e-996d-bc147db7af8b" containerName="payload-pre-processing"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454401    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="221dd229-4d95-4c4e-996d-bc147db7af8b" containerName="payload-pre-processing"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454454    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e2b429-211b-48b7-9460-0fd73202bef3" containerName="maas-api"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454461    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4c3291d-abfd-4f5c-a402-eea21758eafc" containerName="payload-pre-processing"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454468    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454474    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454480    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerName="istio-proxy"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454486    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe8adf2-4fde-4936-917f-7f0dcdd2b071" containerName="istio-proxy"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454492    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="31b357f0-1a81-4d91-9ed2-10db5b9f4a29" containerName="istio-proxy"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454498    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="af975e5f-aa5e-4237-9818-1c8a6f134748" containerName="payload-pre-processing"
Jul 31 22:35:15.454496 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454504    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d857d9e1-caf3-42a1-b68e-5a6e9427e23b" containerName="istio-proxy"
Jul 31 22:35:15.455320 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454512    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerName="istio-proxy"
Jul 31 22:35:15.455320 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454521    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="221dd229-4d95-4c4e-996d-bc147db7af8b" containerName="payload-pre-processing"
Jul 31 22:35:15.455320 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.454530    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="527acc4e-c0bb-4f02-9bf0-3bd71e5598ad" containerName="istio-proxy"
Jul 31 22:35:15.457421 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.457405    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.459269 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.459246    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pm8cl\"/\"kube-root-ca.crt\""
Jul 31 22:35:15.459387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.459297    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pm8cl\"/\"openshift-service-ca.crt\""
Jul 31 22:35:15.459387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.459255    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pm8cl\"/\"default-dockercfg-cz62l\""
Jul 31 22:35:15.464490 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.464409    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"]
Jul 31 22:35:15.602654 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.602616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-lib-modules\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.602654 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.602650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-podres\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.602874 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.602680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-sys\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.602874 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.602777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-proc\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.602874 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.602812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99h2m\" (UniqueName: \"kubernetes.io/projected/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-kube-api-access-99h2m\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.703475 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.703436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-lib-modules\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.703475 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.703476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-podres\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.703694 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.703522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-sys\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.703694 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.703614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-proc\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.703694 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.703630    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-podres\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.703694 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.703660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99h2m\" (UniqueName: \"kubernetes.io/projected/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-kube-api-access-99h2m\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.703830 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.703697    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-sys\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.703830 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.703753    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-proc\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.703830 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.703797    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-lib-modules\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.710681 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.710615    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99h2m\" (UniqueName: \"kubernetes.io/projected/27ede1b6-e8eb-4544-8cdc-2648b78d6fc3-kube-api-access-99h2m\") pod \"perf-node-gather-daemonset-75s2z\" (UID: \"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3\") " pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.767903 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.767871    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:15.901103 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.901042    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"]
Jul 31 22:35:15.903900 ip-10-0-135-241 kubenswrapper[2561]: W0731 22:35:15.903867    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod27ede1b6_e8eb_4544_8cdc_2648b78d6fc3.slice/crio-1b42a4635e9aefbf59a1cef26848da2cda24555d3560184374a0c2291edbf5ca WatchSource:0}: Error finding container 1b42a4635e9aefbf59a1cef26848da2cda24555d3560184374a0c2291edbf5ca: Status 404 returned error can't find the container with id 1b42a4635e9aefbf59a1cef26848da2cda24555d3560184374a0c2291edbf5ca
Jul 31 22:35:15.917316 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.917289    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/2.log"
Jul 31 22:35:15.926628 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:15.926600    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-4wqtz_12a7ea88-8e73-44af-982d-adc960254ccc/console-operator/3.log"
Jul 31 22:35:16.670060 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:16.670020    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z" event={"ID":"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3","Type":"ContainerStarted","Data":"a8b11f12906d8b3a8de03a8bca8b93f3f4a1165cad04861327f986bb9c4d97db"}
Jul 31 22:35:16.670492 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:16.670083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z" event={"ID":"27ede1b6-e8eb-4544-8cdc-2648b78d6fc3","Type":"ContainerStarted","Data":"1b42a4635e9aefbf59a1cef26848da2cda24555d3560184374a0c2291edbf5ca"}
Jul 31 22:35:16.670492 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:16.670111    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:16.684411 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:16.684356    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z" podStartSLOduration=1.684337916 podStartE2EDuration="1.684337916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:35:16.683380174 +0000 UTC m=+3556.946128539" watchObservedRunningTime="2026-07-31 22:35:16.684337916 +0000 UTC m=+3556.947086282"
Jul 31 22:35:18.148583 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:18.148544    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xbrvr_7360b4d5-6dd2-467d-a3c8-75167012a942/dns/0.log"
Jul 31 22:35:18.165616 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:18.165593    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xbrvr_7360b4d5-6dd2-467d-a3c8-75167012a942/kube-rbac-proxy/0.log"
Jul 31 22:35:18.187861 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:18.187835    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d2sf8_cf6f7a4b-33a6-4f10-8e3d-c70fa4e0119d/dns-node-resolver/0.log"
Jul 31 22:35:18.791874 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:18.791840    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sgfk2_00bd2ff5-1c1e-4ba2-8a4d-353cab6f3d3e/node-ca/0.log"
Jul 31 22:35:19.873724 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:19.873695    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64667dffd7-bl6qh_3963e755-e179-43ca-9d44-0a1c1dd6e0bd/kube-auth-proxy/0.log"
Jul 31 22:35:19.968387 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:19.968361    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-z4jdf_2b12a163-fd17-4b59-a3a4-c9397968d10a/payload-pre-processing/0.log"
Jul 31 22:35:20.007402 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:20.007374    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-87xjd_6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4/payload-processing/0.log"
Jul 31 22:35:20.835578 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:20.835545    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r9jbm_7ce0cdef-4292-4fa8-b7ff-c7fba201bbae/serve-healthcheck-canary/0.log"
Jul 31 22:35:21.329525 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:21.329491    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5v7cq_dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d/kube-rbac-proxy/0.log"
Jul 31 22:35:21.347977 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:21.347952    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5v7cq_dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d/exporter/0.log"
Jul 31 22:35:21.366434 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:21.366412    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5v7cq_dc6066a1-12cb-4ccd-b766-bf6c54ed1a0d/extractor/0.log"
Jul 31 22:35:22.684614 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:22.684585    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pm8cl/perf-node-gather-daemonset-75s2z"
Jul 31 22:35:24.093273 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:24.093241    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79f8b57f55-dm5zm_a98c3c97-149b-4886-aa32-8708d6f1a775/manager/0.log"
Jul 31 22:35:24.346974 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:24.346947    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67685b9c4d-gnpfk_e837a855-7075-4797-8c7e-a8ac8f2ab514/manager/0.log"
Jul 31 22:35:24.363650 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:24.363626    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67685b9c4d-gnpfk_e837a855-7075-4797-8c7e-a8ac8f2ab514/copy-manifests/0.log"
Jul 31 22:35:25.622258 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:25.622225    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64667dffd7-bl6qh_3963e755-e179-43ca-9d44-0a1c1dd6e0bd/kube-auth-proxy/0.log"
Jul 31 22:35:25.686490 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:25.686434    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-z4jdf_2b12a163-fd17-4b59-a3a4-c9397968d10a/payload-pre-processing/0.log"
Jul 31 22:35:25.715225 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:25.715192    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-87xjd_6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4/payload-processing/0.log"
Jul 31 22:35:25.756998 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:25.756970    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bf58c775b-5drkm_43e7bdfd-3aef-44a0-b785-20f877f0da2f/manager/0.log"
Jul 31 22:35:25.816713 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:25.816678    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-tl9nv_f125d920-3042-4343-9b77-a348f29a4dc9/openshift-lws-operator/0.log"
Jul 31 22:35:27.312788 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:27.312741    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64667dffd7-bl6qh_3963e755-e179-43ca-9d44-0a1c1dd6e0bd/kube-auth-proxy/0.log"
Jul 31 22:35:27.386968 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:27.386931    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-z4jdf_2b12a163-fd17-4b59-a3a4-c9397968d10a/payload-pre-processing/0.log"
Jul 31 22:35:27.416293 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:27.416259    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-87xjd_6ac7f28e-9e4c-4ff1-8dc3-bd995648ddd4/payload-processing/0.log"
Jul 31 22:35:30.495158 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:30.495103    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-v82d4_7cd3c871-741b-4e2c-949f-c2efa27f51b4/migrator/0.log"
Jul 31 22:35:30.512660 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:30.512622    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-v82d4_7cd3c871-741b-4e2c-949f-c2efa27f51b4/graceful-termination/0.log"
Jul 31 22:35:30.835440 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:30.835406    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-v7rql_26a15230-5d14-464e-bc26-9df69a8d5967/kube-storage-version-migrator-operator/1.log"
Jul 31 22:35:30.837288 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:30.837259    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-v7rql_26a15230-5d14-464e-bc26-9df69a8d5967/kube-storage-version-migrator-operator/0.log"
Jul 31 22:35:31.939560 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:31.939531    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk845_9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc/kube-multus-additional-cni-plugins/0.log"
Jul 31 22:35:31.957447 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:31.957420    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk845_9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc/egress-router-binary-copy/0.log"
Jul 31 22:35:31.975620 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:31.975593    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk845_9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc/cni-plugins/0.log"
Jul 31 22:35:31.993845 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:31.993816    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk845_9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc/bond-cni-plugin/0.log"
Jul 31 22:35:32.011501 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:32.011470    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk845_9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc/routeoverride-cni/0.log"
Jul 31 22:35:32.030898 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:32.030869    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk845_9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc/whereabouts-cni-bincopy/0.log"
Jul 31 22:35:32.053308 ip-10-0-135-241 kubenswrapper[2561]: I0731 22:35:32.053279    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk845_9feb5b8a-a382-4bb3-8a58-0f6fa2a0edcc/whereabouts-cni/0.log"